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U.S. efforts to reform mathematics and science education. The United States 
Congress authorized the Regional Consortia Program in 1990 under the 
Excellence in Mathematics, Science, and Engineering Education Program. In 
reauthorizing the Regional Consortia Program, Congress reaffirmed the purpose 
of the Consortia supported by the program as being to disseminate exemplary 
mathematics and science education instructional materials and assist school 
staff in the implementation of teaching methods and assessment tools. The 
data presented in this report were collected largely in the spring of 1996 
through surveys, interviews, site visits, and review of a variety of 
documents. Another more limited round of data collection was completed in 
spring 1997. This included telephone interviews and document review. Chapters 
include: (1) "Introduction"; (2) "Evolution of the Regional Consortia Program 

and the Challenge of Working with Intensive Sites"; (3) "Professional 
Development"; (4) "Teams and Networks: Consortia Links To the Field"; (5) 
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"Networking and Coordination"; and (8) "Conclusions". Appendices contain the 
survey procedures, instruments, and program indicators. (ASK) 
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EXECUTIVE SUMMARY 



This report analyzes the contributions that the 10 Eisenhower Regional 
Mathematics and Science Education Consortia have made to the nation’s efforts to 
reform mathematics and science education. The U.S. Congress authorized the Regional 
Consortia Program in 1990 under the Excellence in Mathematics, Science, and 
Engineering Education Act (P.L. 101-589, Title II). The program was reauthorized under 
the Improving America’s Schools Act of 1994 (P.L. 103-382, Title XIII, Part C). In 
reauthorizing the Regional Consortia Program, Congress reaffirmed the purpose of the 
Consortia supported by the program as being: (1) to disseminate “exemplary mathematics 
and science education instructional materials” and (2) to assist school staff in “the 
implementation of teaching methods and assessment tools.” Both the original statute and 
the reauthorizing legislation required each Consortium to establish a regional board to 
“oversee the administration and establishment of program priorities....” The statutes also 
went on to list a large number of activities that the Consortia could engage in to achieve 
the two broad purposes specified by Congress. Subsequently, the Department of 
Education set several priorities that helped define expectations for the work of the 
Consortia. In the first grant period, the Department’s solicitation for proposals set an 
absolute priority on assistance in systemic reforms in mathematics and science education 
and assistance to states. In the second grant period, the Department’s solicitation 
included an invitational priority on intensive services to selected local sites. 

The data presented in this report were collected largely in the spring of 1996 
through: (1) surveys of approximately 1,000 participants and recipients of Consortia 
activities and services; (2) interviews with Consortia staff, participants in Consortia 
activities, and others who were familiar with the Consortia’s work; (3) site visits; and 
(4) review of a variety of documents, including Consortia proposals, refunding requests, 
reports, and products. Another, more limited, round of data collection was completed in 
spring 1997. This included telephone interviews and document review. In addition to 
this report and the two rounds of data collection activities, the 4-year national evaluation 
of the Regional Consortia Program has included a round of data collection completed in 
1994 and an interim report, which was completed in 1995. 
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Evolution of the Consortia Program 

The Regional Consortia Program, which is housed in the Department’s Office of 
Educational Research and Improvement (OERI), supports 10 Consortia. Eight of the 10 
current Consortia have received support since October 1992. The two Consortia that won 
grants in the most recent competition (September 1995) are the Eisenhower Regional 
Alliance for Mathematics and Science Education and the Science and Mathematics 
Consortium for Northwest Schools. Seven of the current Consortia grants were awarded 
to organizations that also held contracts for regional educational laboratories. In fiscal 
year 1993, the first year of support, funding for individual Consortia ranged from just 
under $1 million to just over $1.55 million. In the second grant period, which began in 
October 1995 and ends in September 2000, each Consortium was funded at 
approximately $1.5 million annually. 

Like any new assistance organizations, the Consortia faced several challenges 
during their early years of operation. They had to build their organizational capacity and 
market themselves as potentially valuable resources and allies in a complex and not 
always welcoming reform arena. They also had to find ways of collaborating with each 
other so that the Regional Consortia Program and the Eisenhower National Clearinghouse 
(ENC) would become greater than the sum of the individual parts. 

By the end of the first grant period, the Consortia had generally succeeded in the 
tasks of building institutional capacity and establishing themselves as players in the 
mathematics and science education reform arena. They had also learned important 
lessons about the strategies and benefits of working with each other, with the ENC, and 
with other entities in their regions. Finally, as the evaluators reported in the First Interim 
Report on the national evaluation, the Consortia also had to face the basic issue of how to 
invest relatively modest resources to have the greatest impact on large and complex 
problems. 

Findings about Consortia Activities and Services 

This evaluation focused on six categories of Consortia activities and services. The 
categories include professional development, support for teams and networks, 
dissemination of a volume on promising practices in mathematics and science education, 
and intensive work with selected sites. In addition, the evaluators reviewed the 
Consortia’s substantive focus on technology and their use of technology to augment other 
assistance to the field. Finally, the evaluators'examined the collaborative work among 
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the Consortia, the ENC, and other entities — including mathematics and science reform 
initiatives in their regions as well as nationally. 

Across all the events, products, and working relationships for which the evaluation 
team collected data in the spring of 1996 and 1997, participants and others who were 
familiar with the work of the Consortia generally praised their activities. Substantial 
numbers credited their participation in Consortia activities with making at least a 
moderate contribution to their knowledge, skills, and professional activities. In many 
instances, participants called the contribution “major” with regard to knowledge, skills, 
and individual or organizational behavior. In other cases, the Consortia added value by 
increasing opportunities for participants to communicate and collaborate with colleagues 
and peers who shared common concerns and tasks. At the institutional level, they added 
value by supporting or facilitating the next step in a project — sometimes one that was not 
in the original plan or budget but that became important later as the work progressed. 

Intensive Work with Selected Sites 

This work, which in the current grant period is conducted by six of the Consortia in 
response to an invitational priority set by the Department, entails concentrating resources 
and other assistance in a small number of schools, school districts, or other entities. 
Services to these sites included direct technical assistance, professional development, 
financial support, and, in some cases, research and evaluation assistance. However, 
packages of assistance varied from one Consortium to the next. By the spring of 1 997, 
several important lessons had emerged from the Consortia’s experiences in these sites: 

• Careful targeting and reaching agreement on long-term efforts and availability 
of other resources have obvious payoffs, as does the Consortia’s ability to 
broker additional resources from other institutions. 

• Sites must make a clear commitment to working on specific improvement tasks 
and to work with the Consortia and other sources of assistance. 

• As regional entities, the Consortia must strike a balance between the large, 
concentrated resource investments required for effective work in the intensive 
sites and regional service and visibility. 

Professional Development 

Consortia professional development activities reviewed in the evaluation include 
events of several days’ duration for various regional groups (often with longer-term 
follow-up); programs of local mini-grants and stipends to local groups and individual 
teachers and other educators; and intensive, sustained work with small groups of teachers 



and others at selected sites, including some that the Consortia describe as “intensive 
sites.” Participants in these professional development activities generally praised the 
Consortia for the quality of these offerings and the contributions to skills and knowledge. 
Most participants surveyed said that they were doing something differently as a result of 
the professional development they received. In addition, the evaluators found that: 

• The Consortia tended to achieve the most impressive results in professional 
development when they worked with individuals, and, especially, with 
organizations that had a commitment to do something — to deliver services to 
others or to engage in organizational change. 

• Most participants found a good fit between the activities’ purposes and their 
professional responsibilities and existing capacities. 

• Long-term, direct work with groups of teachers stimulated professional 
conversations and experimentation with new classroom practices, but more 
depth and persistence are probably necessary for teachers to change their 
classroom instruction — persistence that may be difficult to achieve, given 
fluctuations in participation by some teachers in some sites. 

Support for Teams and Networks 

The Consortia have convened and supported state teams and regional networks to 
establish links to the field and to provide opportunities for educators to work together on 
common tasks. Members of these groups worked in colleges and universities, state 
education agencies, intermediate education agencies, local school district offices, schools, 
and other organizations. Many were grantees of federally funded programs, such as the 
National Science Foundation’s Statewide Systemic Initiatives and the Department’s 
Eisenhower State Curriculum Frameworks Projects, and most were involved in efforts to 
reform mathematics and science education. The Consortia provided financial and 
logistical support to the teams and networks and they made substantive contributions to 
these groups’ myriad activities. Members of the teams and networks reported that they 
valued their participation in these groups for providing opportunities to open new lines of 
communication with individuals active in reforms whom they might not otherwise have 
met, to discuss important issues, and to forge new working relationships within their 
states and across their regions. The evaluators also found that shifting priorities within 
states and the presence or absence of major reform initiatives greatly influenced the work 
of these groups, especially the state teams. 

Other, specific findings about the Consortia’s support of team and networks 
included: 
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• A large majority of members said that the work of the teams and networks was 
closely related to important reform efforts. 

• Significant numbers of the participants, particularly those on the state teams, 
reported that the activities supported by these groups represented important 
contributions to reforms in mathematics and science education. 

• From the Consortia’s perspective, working with and through the teams helped 
connect them to the field and enhanced their credibility as sources of support 
and assistance. 

Dissemination of a Volume on Promising Practices 

As part of their effort to respond to the charge to disseminate exemplary 
mathematics and science education instructional materials, the Consortia jointly 
developed Promising Practices in Mathematics and Science Education and published it 
in 1995. This volume, which was widely disseminated by the Consortia, provides brief 
descriptions of promising programs and practices. The surveys indicate that: 

• Recipients who remembered receiving the book, 62 percent of those surveyed, 
found the book reasonably useful for the purposes of gathering information or 
staying up to date professionally. 

• Recipients reported little change in behavior following receipt of the book. 

The evaluators concluded that as a dissemination activity, the distribution of the 
book was successful. However, the evaluators also concluded, along with the Consortia, 
that, absent any follow-up support for implementation, this kind of activity is unlikely to 
have much impact on reform efforts. 

Use of Technology 

In general, the Consortia went beyond their mandate and used technology in a 
variety of ways to achieve the purposes of the program. In examining the Consortia’s use 
of technology, the evaluators found that: 

• All Consortia had established World Wide Web sites, but the sites varied in 
sophistication and the amount of information they contained. 

• Several Consortia facilitated the work of teams, networks, and other groups by 
establishing and maintaining electronic networks among the members. 

• The Consortia were involved, to varying degrees, in providing professional 
development, technical assistance, and publications that promoted the use of 
technology in education, particularly the use of the Internet as a resource for 
communication, professional development, and instruction. 
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Networking and Coordination 

This evaluation also reviewed the Consortia’s ongoing networking and coordination 
efforts, including the development of a set of performance indicators, which was required 
under the Government Performance and Results Act. The evaluation team concluded that 
the Consortia’s efforts to work together, with the ENC, and with others have had some 
important payoffs, including the following: 

• Cross-Consortia task forces that developed products for use by all the Consortia 
helped achieve economies of scale and allowed all of them to capitalize on the 
strengths of individual Consortia. 

• The development of the Cross-Consortia Descriptive Data System and program 
performance indicators have the potential to contribute to the Consortia’s 
capacity to reflect on, describe, and assess the quality and impact of their work, 
and will help the Department monitor the work of the individual Consortia, as 
well as the Regional Consortia Program itself. 

• Collaboration with the ENC became a fairly routine part of Consortia operations 
and produced benefits in dissemination of a broad range of materials. 

• All the Consortia worked with major federal mathematics and science education 
reform initiatives, including the National Science Foundation’s systemic 
initiatives and the Eisenhower State Curriculum Frameworks Projects. 

• The Consortia’s work with other technical assistance centers, including the 
technology technical assistance centers, the desegregation assistance centers, 
and the comprehensive technical assistance centers, showed several promising 
starts but had not yet matured. 

As in other areas of their work, determining which of these partnerships represents 
the best use of their limited resources has been an important strategic decision for the 
Consortia. The Consortia have had to weigh these opportunities against each other as 
well as other options they have before investing resources, and they have had to assess 
the impact of their contributions once the investments were made. 

Concluding Observations and Challenges for the Future 

Looking back over 5 years of institutional development and services, this evaluation 
concludes that Consortia have added value to the nation’s mathematics and science 
education reform initiatives. A second conclusion is that the Consortia have matured as 
organizations, both in terms of their capacity to provide a variety of services and to work 
in concert with a variety of other players in mathematics and science reform. 

Accordingly, since the early years of the program, the individual Consortium portfolios 
have taken on a clearer focus. By the spring of 1997, the Consortia’s portfolios of 



activities and services had become relatively stable, changing only as individual tasks and 
activities were completed and others were initiated either in response to requests from the 
field or to invitations and requests from the Department. 

The continued development of the Consortia reporting system and of the Regional 
Consortia Program’s performance indicator system have the potential of adding more 
clarity and focus to the program’s purpose, particularly as future national evaluations and 
the Consortia’s own internal evaluations report on the Consortia’s performance against 
the new performance indicators. 

Clarity of purpose and focus is critical because the Regional Consortia Program and 
the individual Consortia continue to confront the difficult choices inherent in the omnibus 
character of the program’s charge and priorities from OERI and Congress, as well as the 
diverse — and sometimes changing — needs and priorities of their regions. The Consortia 
have tried to do many things, but they are not big enough — in terms of staff and 
resources — individually or collectively, to do everything in improving mathematics and 
science education. At the same time, the program is neither small enough nor yet secure 
enough to pick a tightly specified mission and stick to that mission. To the extent that 
this tension informs an ongoing dialogue about what the Consortia can and should do and 
guides thoughtful decisions about priorities, it will have a beneficial effect on the 
program. 

Thus, the overall value added and obvious maturation of the individual Consortia 
and the Regional Consortia Program as a whole notwithstanding, the final conclusion of 
this evaluation is that important challenges remain — challenges to be addressed by the 
Consortia, by OERI, and by Congress as they plan for the future of the program. When 
viewed in the context of the ambitious goals for the Regional Consortia Program, the 
current annual funding level of $15 million looks like a relatively modest investment in 
solutions to large and complex issues. Consortia collaborations to achieve economies of 
scale and to take full advantage of the special capacities of individual Consortia coupled 
with assistance strategies that leverage resources from other institutions and programs 
have helped stretch these limited resources. These observations as well as similar 
observations about elements of the Consortia’s work point to a recurring theme in the 
evaluation findings: the importance of making strategic choices about investing limited 
resources to achieve maximum impact. Therefore, it is incumbent on both OERI and the 
Consortia to continue seeking clarity about the program’s mission and how well that 
mission is carried out; to set appropriate expectations for the quality, scope, outcomes, 
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and impact of the Consortia’s portfolios of activities and services; and to monitor the 
extent to which those expectations are met. 
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I. INTRODUCTION 



This report analyzes the contributions that the Eisenhower Regional Mathematics 
and Science Education Consortia have made to the nation’s reform of mathematics and 
science education. Congress first authorized the Regional Consortia Program in 1990 
under the Excellence in Mathematics, Science, and Engineering Education Act (P.L. 101- 
589, Title II). The program was reauthorized under the Improving America’s Schools 
Act of 1994, IASA or (P.L. 103-382, Title XIII, Part C). The reauthorizing legislation 
reaffirmed the Consortia’s two main functions: 

• Disseminating exemplary mathematics and science education instructional 
materials. 

• Providing technical assistance for the implementation of teaching methods and 
assessment tools for use by elementary and secondary school students, teachers, 
and administrators. (Sec. 13301) 

In addition to carrying out these two broad statutory functions, the Consortia have 
had to respond to a number of other charges during the past 5 years. These included a 
lengthy list of authorized activities and invitational priorities requiring the Consortia to 
work in concert with other initiatives addressing standards-based systemic reform and 
with various state and local programs. In addition, the Consortia responded to invitations 
from the U.S. Department of Education (ED) to conduct a variety of short-term projects. 
Consequently, the Consortia provide an array of services to states, federally funded and 
privately funded initiatives, school districts, schools, and community organizations. They 
offer technical and financial assistance, professional development, dissemination, and 
forums for communication. 

The Regional Consortia Program supports 10 Consortia (Exhibit 1). Eight of the 10 
current Consortia have received support since October 1992. The two new Consortia that 
won grants in the most recent competition (September 1995) are the Eisenhower Regional 
Alliance for Mathematics and Science Education Reform (Regional Alliance) and the 
Science and Mathematics Consortium for Northwest Schools (SMCNWS). Seven of the 
current Consortia grants were awarded to organizations that also held contracts for 
regional educational laboratories. In fiscal year 1993, the first year of support, funding 
for individual Consortia ranged from $998,401 to $1,550,899, with most of the Consortia 
receiving between $1.1 million and $1.2 million. In the second grant period (beginning 
October 1995), each Consortium was funded at $1.5 million annually for 5 years. 
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Exhibit 1 EISENHOWER REGIONAL CONSORTIA PROGRAM 



When viewed in the context of the ambitious goals that were established for the Regional 
Consortia Program, these funding levels look like relatively modest investments in 
solutions to large and complex issues. 

Evaluation Methods 

This report presents findings from all four years of the evaluation. However, given 
that the evaluation’s most intensive data collection activities occurred during the spring 
of 1996, this report relies most heavily on data from that period. More recent information 
gathered during the spring of 1997 is highlighted throughout the report. 

Spring 1996 data collection was designed to elicit participants’ views on the 
outcomes or attributes of the following categories of Consortia activities: 

• Professional development, because skill building for the field is a vital 
component of any technical assistance program. 

• Support for teams and networks, which has been a distinctive and apparently 
resource-intensive strategy of this program across many (although not all) of the 
Consortia. 

• Dissemination of information about promising practices, which responds 
directly to a key part of the legislative charge. 

Although only a portion of the Consortia’s activities and services, these categories and 
the individual activities within them represented important elements of the Consortia’s 
work. This evaluation examined mainly those activities that in spring 1996 were (1) 
relatively mature, (2) recently completed, (3) relatively large investments of Consortia 
resources, or (4) key parts of individual Consortia portfolios. 

In addition, the evaluation looked at how the Consortia have used technology for 
dissemination, networking, training, and technical assistance activities. Finally, the 
evaluation examined the Consortia’s own coordination and networking activities because 
the Consortia are required to work with each other, the Eisenhower National 
Clearinghouse (ENC), other technical assistance providers, and mathematics and science 
education reform initiatives. 

Data collection activities completed in the spring of 1996 included the following: 

• Customized surveys administered by mail to participants (or recipients of a 
product) in each category of Consortia activities. (Copies of the surveys and a 
summary of the procedures are included in Appendix A.) 



® On-site and telephone interviews with participants and with Consortia directors 
and staff. 

• Observations of a small subset of activities. 

• Telephone interviews with state and local program administrators who were 
familiar with the work of the Consortia. 

• Review of documents (e.g., grant proposals, annual reports and refunding 
requests, Consortia products, internal evaluation reports). 

Data collection activities completed in the spring of 1997 were intended to update 
information on the Consortia’s overall portfolios of activities and services, including their 
working relationships with other federal mathematics and science education reform 
initiatives and technical assistance centers, to review Consortia mechanisms for quality 
control, and to review progress in work with “intensive sites.” To accomplish these 
goals, spring 1997 data collection typically included telephone interviews with Consortia 
directors; professional staff, including the Consortia staff evaluators; and a member of 
each Consortium’s regional advisory board or other key advisory group, such as a state 
team. In addition, the evaluators reviewed a variety of documents, including the annual 
refunding requests submitted by the Consortia in the spring of 1997. 1 

Organization of the Report 

Following this introduction, the report begins with a brief overview of the evolution 
of the Regional Consortia and an examination of the Consortia’s recent work with 
intensive sites. The next three sections present, in turn, findings about the effectiveness 
and quality of selected Consortium activities in (1) professional development, (2) support 
for teams and networks, and (3) dissemination of information about promising practices 
in mathematics and science education. The report then discusses the Consortia’s use of 
technology to augment their services, as well as their efforts to promote the use of 
technology in improving mathematics and science education generally. Following that, 
there is a review of the Consortia’s collaborative and networking activities, highlighting 
the development of the program performance indicators and the Consortia’s work with 



' At ED’s direction, this report combines the Second Interim Report, completed by the contractor in 1996, 
and the Final Summary Report, originally scheduled for completion in 1997. ED’s request that this 
report combine the Second Interim Report and the Final Summary Report came after planning and data 
collection in the spring of 1997 were completed. Therefore, data collection during that period did not 
specifically focus on following up on all activities that were in progress during the earlier round of data 
collection. Wherever possible, this report does update earlier data. 
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other technical assistance providers and major mathematics and science education reform 
initiatives. The last section of the report contains the evaluation team’s overall 
conclusions about the evolution and work of the Consortia during the period covered by 
this evaluation. It also comments on directions for the continued development of the 
Regional Consortia Program. 
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II. EVOLUTION OF THE REGIONAL CONSORTIA PROGRAM AND THE 
CHALLENGE OF WORKING WITH INTENSIVE SITES 



Evolution of the Regional Consortia Program 

When the Consortia first began operations in 1992, they faced a number of 
challenges. To meet these challenges, they had to build their organizational capacity and 
market themselves as potentially valuable resources and allies, so that when they came to 
the reform table, they had something to contribute. They also had to find ways of 
collaborating with each other so that the Regional Consortia Program would become 
greater than the sum of the individual parts. In the First Interim Report of this evaluation, 
the evaluation team described the challenges facing the Consortia as follows: 

First, and perhaps most important, they [the Consortia] were asked to fulfill an 
ambitious agenda ranging from giving direct assistance to classroom teachers to 
supporting states in systematically reforming their entire educational systems. 
Indeed, many of the specific tasks they were asked to take on — implementing new 
approaches to preservice education, for example — posed technical and political 
problems that many of the nation’s best minds and institutions have been struggling 
with for decades. ... Second, among the key targets of their assistance were state- 
level entities with established norms, political agendas, and technical capacities. 
Finally, by focusing on standards-based reforms, the Consortia were entering an 
already-crowded reform arena. (Haslam, Colopy, & Turnbull, 1996, p. 10) 

By the end of the first grant period, the Consortia had generally succeeded at the 
tasks of building institutional capacity and establishing themselves as players in the 
mathematics and science education reform arena. They had also learned important 
lessons about the strategies and benefits of working with each other, with the Eisenhower 
National Clearinghouse, and with other entities in their regions. Finally, as the evaluation 
team observed in the conclusions to the First Interim Report: 

One of the basic dilemmas the Consortia face is whether to invest their relatively 
modest resources in activities that reach large numbers of people in limited ways or 
concentrate on serving a relatively small number of people and programs more 
intensively. The overall portfolio of Consortia activities reflects efforts to do both. 
(Haslam, Colopy, & Turnbull, 1996, p. 58) 

As this report suggests in several places, the Consortia continue to face this dilemma and 
the attendant changes of making sound strategic choices about how to invest their 
resources. 
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The two new Consortia funded in 1995 faced similar challenges. They had to build 
new organizations and establish working relationships with other organizations in their 
regions. Nonetheless, they had one advantage: they could draw on the experience of the 
other eight Consortia. Evaluation data from the spring of 1996 suggest that the Regional 
Alliance was able to move ahead with its work and that the Science and Mathematics 
Consortium for Northwest Schools (SMCNWS) was less successful. Subsequently, and 
with considerable support and encouragement from OERI and other Consortia, 

SMCNWS has gradually developed its capacity and established reasonably solid working 
relationships with the other Consortia and with most of its regional stakeholders. The 
progress that this Consortium made is evidence of the growing strength of the Consortia 
network and the partnership between the Consortia and OERI. 

From October 1995 through the spring of 1997, the program portfolio of activities 
and services, as well as the portfolios of individual Consortia, generally remained stable 
in terms of the types of clients served and the strategies the Consortia used to serve them. 
Thus, the Consortia continued to provide a variety of professional development, training, 
and technical assistance at both the state and local levels. They supported a number of 
state teams and regional networks that brought together regional, state, and local leaders 
in mathematics and science education to discuss issues of common concern and to advise 
the Consortia about their needs for assistance and support. The teams and networks also 
used Consortia resources to organize and support state and regional projects. The 
Consortia, often in collaboration with other Consortia and the ENC, produced and 
disseminated an array of materials related to improving mathematics and science 
education. They also disseminated materials developed by other sources — often as part 
of Consortia support for individual mathematics and science education reform initiatives. 
These materials included the Consortia’s newsletters, descriptions of what the Consortia 
call promising and exemplary practices, and guides to resources that assist educators in 
their reform efforts. Consortia reports indicate that these products have reached 
thousands of people across the country. Using many of these same assistance strategies, 
the Consortia worked with major state and local mathematics and science education 
reform initiatives. 

While the overall portfolio of Consortia activities and services remained relatively 
stable, individual Consortium activities waxed and waned as staff completed some tasks 
and replaced them with others. For example, as the Statewide Systemic Initiatives (SSIs) 
supported by the National Science Foundation began to complete their work, the 
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Consortia began to redirect their efforts to the Urban Systemic Initiatives (USIs) and 
Rural Systemic Initiatives (RSIs), both of which started more recently. To illustrate: 
work with two RSIs that focused, in part, on improving the education of Native American 
students led the High Plains Consortium to increase attention to this area in its portfolio 
of work. 

Responses to invitations from ED, such as invitations to assist in pilot testing new 
criteria for identifying promising and exemplary practices or in communicating 
information about the Third International Mathematics and Science Study (TIMSS), 
required bursts of activity and some shifts in staff assignments and resource allocations. 
These did not fundamentally alter overall directions and service delivery strategies. 

Throughout the evaluation, one of the most difficult challenges facing the 
evaluation team has been to analyze and describe the work of the Consortia clearly and 
comprehensively. The difficulty of capturing an overall sense of the Consortia’s work 
mirrors a real policy problem facing the Regional Consortia Program. In the beginning, 
the program lacked a tight formulation of its purpose and methods. To some degree, this 
lack of focus was inevitable in a technical assistance program, which had to retain its 
capacity to improvise as opportunities warranted. To the credit of ED and the grantees, 
by the spring of 1997, a clearer program formulation was beginning to evolve. Beginning 
in the spring of 1996, in compliance with the Government Performance and Results Act, 
ED and the Consortia worked together to develop performance indicators that succinctly 
framed the program’s intentions and against which its successes and failures could be 
measured. The final draft of the performance indicators was submitted to the Acting 
Deputy Secretary of Education in September 1997. (The draft of the performance 
indicators is included as Appendix B.) These indicators and subsequent reports on 
Consortia performance that address the indicators should yield a clearer sense of program 
purposes and accomplishments. In the meantime, findings from this evaluation can help 
point to fruitful directions for the Consortia’s initiatives. 

Work with Intensive Sites 

The solicitation for the second grant period (beginning in October 1995) called on 
the Consortia to “focus their activities to achieve maximum impact.” To do so, they were 
expected to provide “training and assistance to classroom teachers, administrators, and 
other educators to enable them to instruct other teachers, administrators, and 
educators....” In the shorthand of the program, this work became known as work with 



“intensive sites.” The remainder of this chapter describes the Consortia’s early efforts to 
address this priority. 

In response to an ED invitational priority in the second grant period, the Consortia 
have paid increased attention to — and some Consortia have made increased investments 
in — direct services at the local level. OERI staff held out the expectation that, in addition 
to making important contributions at the local level, work in the intensive sites would 
result in some institutional learning by the Consortia. 

The Consortia embraced the task of working with intensive sites in quite different 
ways, revealing both the diversity in the program and the very difficult choices that 
organizations with limited resources make in tackling difficult and demanding tasks. In 
the spring of 1996, the evaluators’ review of individual Consortium portfolios showed 
that some of the Consortia had already begun to target at least some of their technical 
assistance and professional development to the local level. Other local-level services, 
however, were just getting under way when the data were collected, and it was too soon 
to assess their impact and effectiveness. 

During the spring of 1997, the evaluation team collected additional descriptive 
information on Consortia activities and services in intensive sites. Five of the 10 
Regional Consortia indicated that they conducted work in intensive sites. Exhibit 2 
summarizes Consortia’s accounts of this work. 

In addition to the five Consortia whose work with intensive sites is summarized in 
Exhibit 2, the High Plains Consortium’s (HPC) Action Research sites, which the chapter 
on professional development in this report also discusses, represent another example of 
work with intensive sites. When data were being collected during the spring of 1997, the 
remaining four Consortia reported that they were not working with intensive sites, 
although several reported that they anticipated doing so in the near future. To be fair, 
there is no agreement about what constitutes intensive sites. All Consortia work on the 
local level. Thus, the evaluation chose to focus on the work of those Consortia who 
declared that they worked with intensive sites. 

Services to intensive sites included direct technical assistance, professional 
development, and, in some cases, research and evaluation assistance and financial support. 
Several Consortia reported that they also brokered additional assistance to these sites. 
Several Consortia also augment their services to individual sites, with large-scale institutes 
that bring the sites, or site representatives, together for training and professional 
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development. Typically, these activities occur in the summer to facilitate participation by 
school staffs. 



Exhibit 2 

REGIONAL CONSORTIA SERVICES TO “INTENSIVE SITES” 



Consortium 


Number of 
Sites 


Selection Criteria 


Strategies 


Regional 

Alliance 


37 schools 


Schools committed to long- 
term reform in math, 
science, and technology 


Schools are members of 
Alliance Schools Network. 
Consortium provides or 
brokers assistance, and 
statewide action team 
identifies critical friends. 
Consortium provides $2,000 
to support schools 
participating in the network. 


Consortium 
at AEL 


Nine schools 


Public and private schools 
with (1 ) at least 60 percent 
of students eligible for free 
or reduced-price lunch or 
tuition, (2) documented 
progress toward 
implementation of school 
improvement plan, and (3) 
leadership committed to 
working with the 
Consortium on a long-term 
plan 


Consortium provides $750 
mini-grant and direct 
assistance in needs 
assessments. Additional 
technical assistance to follow, 
pending availability of 
staff/consultants. Consortium 
also brokers other assistance, 
especially from institutions of 
higher education. 


SERVE 

Consortium 


One region in 
an urban district 


Not applicable 


Consortium provides training 
and technical assistance as 
needed. 


Southwest 

Consortium 

(SCIMAST) 


1 9 district 

professional 

development 

projects 

(included in two 

SCIMAST 

assistance 

programs) 


Site-level commitment to 
long-term, high-quality 
professional development 
to improve mathematics 
and science education 


Consortium provides $20,000 
a year for 5 years for 
Professional Development 
Awards (PDAs) and $1 ,200 a 
year for customized training 
sites. Additional assistance to 
PDA sites includes ongoing 
technical assistance and 
annual meetings of grantees. 


Midwest 

Consortium 


Two to three 
projects per 
state, including 
schools, 
districts, or 
groups of 
school districts 


Commitment to working 
with the Consortium on 
development activities 


Consortium offers sites 
“research support; evaluation 
design; technical assistance, 
planning, sharing 
expertise/best practice, and 
acting as a critical friend and 
capacity builder.” 
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Although the Consortia asserted that the goal of all of this work was to improve 
student learning, their strategies, especially the definitions of “intensive,” were quite 
different. Thus, the professional development grant program operated by the Southwest 
Consortium for the Improvement of Mathematics and Science Teaching (SCIMAST) 
provided substantial long-term support — $20,000 a year for several years — to local 
professional development projects that were explicitly linked to improving instruction. 
This program is discussed in more detail in the chapter on professional development. 

According to staff at the Regional Alliance, schools in the Alliance Schools 
Network had to be committed to long-term improvement efforts, but individual 
improvement goals and strategies varied considerably. Intensive site work by the 
Consortium at AEL included completing a needs assessment for each of nine low- 
performing schools and following up the results with the development of a plan for 
technical assistance that included assistance from the Consortium as well as assistance 
from other sources, including nearby institutions of higher education, state mathematics 
and science reform initiatives, and the comprehensive technical assistance center also 
located at the regional laboratory. 

The Midwest Consortium for Mathematics and Science Education (Midwest 
Consortium) worked with intensive sites (schools, districts, and groups of districts) that 
had already moved toward improvement and were willing to join in what the Consortium 
called a “co-development” effort. In a second example of this Consortium’s intensive site 
work, Consortium staff, with assistance from the North Central Regional Education 
Laboratory, the Consortium’s parent organization, worked with a district team to develop 
a network collaboration project that is intended to result in the implementation of new 
elementary school curricula that incorporate technology. The Consortium anticipates that 
the curricular models that are developed will be available to all 28 of the schools in the 
district. The purpose of these and other co-development activities was to continue work 
on local improvement tasks and to generate products that would be of general use to the 
field. As these and other examples suggest, an important lesson learned from these 
experiences is that there must be clear evidence of commitment on the part of the site — 
both commitment to the specific tasks and commitment to working with the Consortia. 

Consortia directors and staff were generally optimistic about the prospect that these 
activities would pay off for participating sites, and saw their efforts as likely to have 
significant impact. Most also noted the benefits of this work for the Consortia. “On the 
positive side, it provides opportunities for action research, the development of models.” 



The Midwest Consortium’s 1997 refunding request summed up the intended benefits of 
its co-development this way: 

The intensive sites provide a place for action research, product development, field 
testing, dissemination, [development of] craft knowledge, and credibility. The 
wisdom and resources are then shared through workshops and presentations by 
[Midwest Consortium] staff, through trainer-of-trainers efforts by partners, such as 
state departments of education, intermediate units, SSIs, USIs, and district staff. 

Working with intensive sites also posed some challenges for Consortia. As a staff 
member at the Eisenhower Consortium for Mathematics and Science Education at 
SERVE explained: 

Originally in the proposal there were two in each state, but after working with 
[a district] and that being only intensive for a year, we really had to rethink that 
commitment. It takes so much more time than anyone had ever imagined. We 
would rather do less and do a really good job than to spread out and not do a 
good job. 

Directors and staff in other Consortia shared this concern about the costs of working 
in intensive sites. A staff member at the Consortium at AEL talked about a second 
challenge that can emerge in particularly low-performing sites: 

Direct service is a double-edged sword because of the selection criteria.... The 
theory is to pick schools where we could really do good work and that we could 
work with. But in these schools, the people in charge may not care. To latch on to 
a school and make a commitment to it.. .you may have to work more on buy-in. 

They may not want help. I have called some schools at least five times to set up a 
meeting, and only one has responded. 

Working with sites that have little or no progress on their own and that have limited 
capacity to do so, points to the need for substantial investments of time and money just to 
get started. 

A third challenge — and risk — in these long-term, intensive relationships was 
turnover in local staff. In a number of cases, Consortia staff expressed frustration with 
the fact that just as they were building strong working relationships with local leaders and 
staff — a long-term process in itself — these individuals were reassigned or took on other, 
competing responsibilities. “One month we would go to a meeting and agree on some 
plans. When we returned for the next meeting, there was a whole new group of people 
and we pretty much had to start over” is the way one Consortium staff member described 
it. Such turnover has implications for the level of intensity at a site. Although a 
Consortium might continue to work at a high level of intensity, participant turnover may 
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mean that few, if any, participants actually receive intensive services. Indeed, some 
participants may even experience these services as a one-time event. Thus, the overall 
result may be intensive from the Consortium’s perspective but nonetheless diluted from 
the recipient’s perspective. 

Work with intensive sites poses a real dilemma to the Consortia. Experience thus 
far shows that work with intensive sites has the potential for some important payoffs, 
both for the sites and for the Consortia. At the same time, these services are relatively 
expensive, and the Consortia may incur substantial start-up costs as they attempt to build 
long-term partnerships. Their activities are also vulnerable to rapid shifts in local staffing 
and priorities, both of which can seriously undermine local commitment to continue. As 
regional entities, the Consortia must strike a balance between large, concentrated resource 
investments and regional service and visibility. And, as several Consortia are learning, 
their ability to broker additional resources from other institutions has the potential to 
significantly increase the amount of help available in the intensive sites. 
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III. PROFESSIONAL DEVELOPMENT 



Training teachers, administrators, and other educators was a priority in the 
Consortia s statutory list of authorized activities. The original law suggested using a 
training-of-trainers strategy, saying that the Consortia may: 

provide for the training of classroom teachers to enable such teachers to instruct 
other teachers, administrators, and educators in the use of.. .instructional materials, 
teaching methods, and assessment tools. [Sec. 13302 (3)] 

As discussed in the preceding chapter, OERI’s invitational priorities for the second grant 
period advised the Consortia to work in a sustained and intensive way with selected 
schools and districts, saying that this approach reflects research showing the importance 
of a long-term commitment in professional development. As a result, the Consortia 
employed both strategies in an effort to get an effective bang for a limited buck. 

In this chapter, the report examines 1 1 Consortia-sponsored professional 
development activities, characterized in Exhibit 3. The bulk of the data collected for this 
examination — including the surveys, interviews, and a review of any evaluation data 
available from the Consortia— was gathered in the spring of 1996. In addition, reviews of 
annual reports and interviews with Consortia staff conducted in spring 1997 provide 
updated information on a number of these activities. 

Discussions with all the Consortia indicated that these 11 professional development 
activities were the only ones for which any data on effects could reasonably be obtained 
by May 1996. The survey data presented here came from participants in six professional 
development activities. These were the professional development activities for which the 
Consortia provided direct service and had complete participant records. 

The chapter first describes various categories of Consortia-sponsored professional 
development activities. Next, the survey sample is presented and the professional 
development participants are described. This is followed by a discussion of the 
participants’ judgments of the quality of the professional development they received. The 
chapter then discusses the effects of the professional development activities on the 
participants’ skills and knowledge. Next, the chapter presents evidence as to the impact 
of the professional development on the participants’ practice. The chapter concludes with 
a discussion of the strengths and weaknesses of these Consortia activities. 
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Exhibit 3 

OVERVIEW OF PROFESSIONAL DEVELOPMENT ACTIVITIES EVALUATED 



Activity 

(Consortium) 


Purpose/ 

Content 


Consortium 

Role 


Number of 
Participants 


Duration 


Participants’ Job 
Responsibilities 


Participants’ 

Purposes 


Regional 

Training 

Project 

(AEL) 


Train trainers in 

standards-based 

instruction 


Organized 
the event 


350 


2 days 


74 percent 
elementary/ 
secondary 
teaching 


Be resource to others 


TAAMSS 

(SERVE) 


Train leaders in 
reform process 


Organized 
the event 


75 


3.5 days 


72 percent 

professional 

development/ 

technical 

assistance 


Be resource to others; 
strengthen 
organization; engage 
in major change 


TOT 

(SCIMAST) 


Train trainers in 
instruction and 
change 
processes 


Organized 
the event 


36 


4 days 


57 percent local 
administrators 


Engage in major, 
planned organizational 
change 


Celebrating 

Success 

(SERVE) 


Focus on 
promising 
practices and 
change process 


Organized 
the event 


134 


3 days 


62 percent 

professional 

development/ 

technical 

assistance 


Strengthen 

organization; individual 
learning 


Professional 

Development 

Awards 

(SCIMAST) 


Support for long- 
term change 
efforts 


Provided 
grants to 
organizations 


15 grantee 
sites/year 


5 years 


Primarily 
universities and 
school districts 


Change instruction 
through multiyear 
professional 
development 


Participant 

Support 

(SMCNWS) 


Support 
attendance at 
professional 
development 


Provided 
grants to 
teachers 


400 

(summer 

1996) 


Variable 


Primarily teaching 


Individual learning 


Action 

Research 

(HPC) 


Collaborate in 
intensive 
professional 
development 


Organized 

teacher 

study 

groups; 

visited 

groups and 

classrooms 


About 
1 5/site 


Monthly 
meetings 
over school 
year 


Teaching 


Individual learning; 
course credit at one 
site 


Models 

Learning 

Network 

(Midwest) 


Collaborate in 
intensive 
professional 
development 


Organized 
teacher 
study group 


20 


Monthly 
meetings 
over half 
year 


Teaching 


Individual learning and 

strengthening 

organization 


Arizona 

Journey 

Schools 

Program 

(WERC) 


Work with 
schools on 
systemic reform 


Organized 
and led 
series of 
institutes 
and 

workshops 


33 schools 


6 events of 
2-5 days 
over 2 
years 


School teams 


No data from 
participants 


Customized 

Training 

(SCIMAST) 


Help selected 
sites with 
training in a 
topic chosen by 
site, empha- 
sizing construc- 
tivist teaching 


Selected 

sites; 

negotiated 

content; 

made 

multiple 

visits 


Small 
number of 
schools or 
districts 


10 days of 
training 
and follow- 
up over at 
least a year 


Teachers in school 
teams; principal 
must participate 


Limited data from 
participants; individual 
and organizational 
change seem to be 
purposes 


Project Soar 
(SERVE) 


Provide 
intensive 
support to an 
urban subdistrict 


Developed 

local 

leadership 

team; 

helped 

organize 

conference 


20 leaders; 
200 

teachers 


Several 
months so 
far; 5-day 
conference 


Primarily teaching 


Limited data from 
participants; teacher 
leadership in 
instructional change is 
district’s purpose 
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Consortia-Sponsored Professional Development 

Although the categories are not mutually exclusive, the Regional Consortia’s 
professional development activities typically employed a training-of-trainers strategy or 
provided sustained and intensive work with selected sites. The training-of-trainers model 
typically involved professional development events lasting less than a week, and usually 
targeted professional development providers or administrators. Examples of the training-of- 
trainers professional development included: 

• The Regional Training Project of the Consortium at the Appalachia Educational 
Laboratory (Consortium at AEL) offered 2-day sessions for individuals who 
become, in turn, responsible for providing workshops to teachers on the national 
standards in mathematics and science education and ways of implementing these 
standards in classrooms. According to the 1997 annual report of the Consortium at 
AEL, the focus of the Regional Training Project was on the state curriculum 
frameworks in each of the four states in the region. Through this training-of- 
trainers approach, the Consortium intends to provide indirect service to all of the 
region’s 27,500 K-12 mathematics and science teachers. 

• The SERVE Eisenhower Consortium for Mathematics and Science at SERVE’s 3- 
day “Celebrating Success” Leadership Conference, which is part of the Promising 
Practices Leadership Institute, focused on 1 8 promising practices in mathematics 
and science education from the region, with attention to the change processes that 
would be required to implement new practices. The Consortium also describes this 
Institute as a dissemination activity that augments the dissemination of printed 
materials, such as the one discussed in more detail in this report’s chapter on 
dissemination. 

• Technical Assistance Academy for Mathematics and Science Services (TAAMSS), 
offered by the Consortium at SERVE, provided professional development for 
trainers committed to systemic reform. The subject matter was the reform process: 
building support for change; developing local leadership; teaching adults; and 
reform topics such as frameworks, standards, and equity. Each participant was 
required to conduct at least two workshops during the rest of the school year or the 
summer, using the materials from TAAMSS. According to the Consortium’s 1997 
annual report, this focus continued in 1996-97. The Consortium also reported that, 
beginning in May 1997, this activity “will include advanced strategies and tools 
related to reflection, coaching, program monitoring/feedback and providing 
continuous support” and that the training will be augmented by on-site observations 
by Consortium staff. 

As the TAAMSS example suggests, the Consortia have tried to include follow-up and 
more sustained support in the training-of-trainers strategy. The Consortia’s work with 
selected sites represents even more intensive and ongoing support. In contrast to the more 
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standard, large-scale offerings with relatively polished presentations, these activities had a 
more participatory, evolving character. Examples of this approach included: 

• Action Research, conducted by the High Plains Consortium (HPC), operated in two 
sites: St. Louis, where participants were 15 elementary teachers from different 
schools, and a Denver school where 15 teachers joined this study group, which was 
an element of their school improvement plan. Participants discussed issues of 
common concern in small groups, conducted research projects in their classrooms, 
and made presentations to the group. The 1997 annual report indicates that Action 
Research also operates in Wyoming. 

• A more recently formed group was the Models Learning Network (MLN) facilitated 
by the Midwest Consortium for Mathematics and Science Education (Midwest 
Consortium). The MLN met five times between January and June 1996. 

Participants were mathematics and science teachers from elementary, middle, and 
secondary grades in 20 Chicago-area school districts. Between meetings, 
participants were expected to study materials or to prepare presentations. 

At least one Consortium embedded the training-of-trainers strategy in its efforts to 
work intensively with selected sites. SCIMAST’s Professional Development Awards (PDA) 
program provided $20,000 to each of 15 sites, for a total of $300,000. Applicants were 
required to document how they would fulfill several criteria: commit to at least 10 days per 
year of training and follow-up; contribute local resources; involve participating teachers in 
planning; carry out a multiyear plan; emphasize depth over breadth; and focus on 
constructivism, problem solving, hands-on experiences, integrated technology, and alignment 
with standards. SCIMAST offers an annual Training of Trainers (TOT) event to PDA 
participants. The event includes sessions on the process of learning, instructional strategies 
in mathematics, the NCTM standards and instructional alignment, professional development, 
the PDA program, and technology resources. 

Three other Consortia sponsored professional development activities also focused on 
selected sites. Customized Training was an established activity category for SCIMAST. The 
Consortium selected a limited number of sites to work with, requiring each prospective site to 
demonstrate the following: participation by a large number of teachers in each school (as 
opposed to one teacher per school), attendance and support by the principal, 10 days of 
training and follow-up, practice with students, interest in constructivist teaching, and cost 
sharing. Each engagement included a training segment of 5 to 7 days plus two follow-up 
visits with classroom observation and additional training. 

Through Project Soar, the SERVE Consortium assisted one region of Dade County 
(Florida) Schools with what staff called “the works” in assistance and professional 
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development. Beginning in January 1996, this activity focused on developing a 20-member 
leadership team from five feeder patterns and pulling another 200 teachers in school teams in 
these feeder patterns into professional development activities. As noted in the .preceding 
chapter, SERVE’s assistance to this region of the Dade County Schools represents the 
Consortium’s intensive site work. 

The Arizona Journey Schools Program (AJSP) began as an initiative of the Arizona 
Department of Education, with assistance from the WestEd Regional Consortium (WERC). 
When the program’s originators left the state department in the wake of major philosophical 
and organizational changes, WERC moved from a supporting to a leadership role. The 
content of institutes and workshops included systemic change, instruction, assessment, and 
professional development. This activity repeatedly convened school and community teams 
over a 2-year period that began under the Consortium’s first grant. 

In a different approach, the Science and Mathematics Consortium for Northwest 
Schools (SMCNWS), in the first year of its grant, sponsored a program called Participant 
Support, which provided stipends to teachers. It allocated $30,000 to each state in its region 
to underwrite up to a third of the travel costs for individual teachers to attend professional 
development activities. Teachers submitted applications discussing how the activity 
addressed their local reform plan and state goals in mathematics and science. In addition, 12 
workshop organizers received awards of up to $6,000 to expand their summer offerings. An 
example is the arrangements made to allow teachers from four other states to attend summer 
training institutes supported by Idaho’s Eisenhower Professional Development funds. 

Participants in Consortia>Sponsored Professional Development 

In this section, the participant survey results are examined in order to profile who 
participated in the Consortia’s staff development activities. Achieving a good fit between a 
Consortium’s purpose and participants — with respect to the roles they play in their 
organizations and the expectations they bring to professional development activities — is a 
key factor in the success of the activity. To begin, Exhibit 4 displays the sample sizes and 
response rates for participant survey data. 

Each Consortium activity had a distinctive profile with respect to its participants’ 
professional responsibilities (Exhibit 5) and purposes for participating (Exhibit 6). 
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Exhibit 4 



MATHEMATICS AND SCIENCE EDUCATION REGIONAL CONSORTIA PROGRAM 
CUSTOMER SURVEY SAMPLE: PROFESSIONAL DEVELOPMENT 



Professional 
Development Activity 


Consortium 


Total 

Participants 


Sample 

Size 


Surveys 

Returned 


Response 
Rate (%) 


Regional Training 
Project 


Consortium at 
AEL 


370 


49 


39 


80 


Technical Assistance 
Academy for 
Mathematics and 
Science Services 
(TAAMSS) 


SERVE 


77 


38 


28 


74 


“Celebrating Success” 
Leadership Conference 


134 


34 


26 


76 


Training of Trainers 
(TOT) 


SCIMAST 


36 


36 


28 


78 


Action Research 


HPC 


30 


30 


17 


57 


Models Learning 
Network (MLN) 


Midwest 


20 


20 


15 


75 


Total 


667 


207 


153 


74 



Exhibit 5 

RESPONSIBILITIES OF THE PARTICIPANTS IN CONSORTIUM 
PROFESSIONAL DEVELOPMENT 



Responsibilities 


Percentage of Participants 


All 

Participants 


Regional 

Training 

Project 


TAAMSS 


TOT 


Celebrating 

Success 


Action 

Research 


MLN 


K-12 teaching 


62 


82 


48 


35 


36 


100 


82 


Postsecondary 

teaching 


17 


18 


11 


12 


14 


0 


0 


Administering local 
education 
program(s) or 
organization(s) 


23 


18 


33 


54 


27 


0 


9 


Administering state 
education 
program(s) or 
organization(s) 


9 


6 


19 


0 


18 


0 


0 


Professional 
development or 
technical 
assistance 


41 


32 


70 


46 


59 


0 


18 



Note: Columns may not sum to 100 percent because multiple responses were possible. 

Exhibit reads: Sixty-two percent of the participants in professional development activities reported that 
their professional responsibilities included teaching at the elementary or secondary level. 



Source: Customer survey. 
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Exhibit 6 



PURPOSES FOR PARTICIPATING IN CONSORTIUM 
PROFESSIONAL DEVELOPMENT 



Purposes for 
Participating 


Percentage of Participants 


All 

Participants 


Regional 

Training 

Project 


TAAMSS 


TOT 


Celebrating 

Success 


Action 

Research 


MLN 


Keeping current 
professionally 


74 


79 


74 


46 


64 


71 


64 


Obtaining ideas 
or contacts that 
would strengthen 
my work 


77 


79 


78 


62 


73 


74 


82 


Obtaining ideas 
or contacts that 
would strengthen 
my organiza- 
tion’s existing 
work 


54 


41 


81 


58 


77 


31 


55 


Obtaining ideas 
or contacts that 
would help my 
organization 
carry out a 
major, planned 
improvement 
effort 


49 


44 


74 


81 


45 


19 


45 


Being a resource 
to others 


81 


88 


93 


62 


68 


31 


64 



Note: Columns may not sum to 100 percent because multiple responses were possible. 

Exhibit reads: Seventy-four percent of the participants in professional development activities reported 
that at least one purpose for participating was to keep current professionally. 



Source: Customer survey. 



Using the survey data, evaluators constructed rough profiles of the groups participating 
in each of the surveyed activities: 

• TAAMSS participants in the Southeast, compared with participants in other 
activities, were especially likely to be professional developers or administrators of 
state programs. Almost all expected to be “a resource to others,” in the words of the 
survey; most expected to strengthen their organizations or to engage in a major, 
planned organizational change. 

• Of all these activities, Training of Trainers in the Southwest served the highest 
percentage of local administrators, and its participants were the most focused on 
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major, planned changes. (They were presumably referring to the change efforts 
supported by grants from SCIMAST under the Professional Development Awards 
program.) 

• Celebrating Success participants in the Southeast included a fairly high proportion 
of professional developers but were not notable in their expectation of being a 
resource to others or engaging in major change. 

• Regional Training Project participants in the Appalachian region were mostly 
teachers, almost all of whom expected to be a resource to others but whose 
intentions with respect to organizational change were modest. 

• Participants in Action Research with the High Plains Consortium were all teachers. 
Relatively few of them expected organizational improvement or change to ensue 
from their participation in the activity; instead, they focused on the classroom level. 

• Participants in the Models Learning Network in the Midwest were mostly teachers 
who participated to obtain ideas or contacts that would strengthen their work; some 
expected organizational improvements or changes. 

Although their profiles varied in a number of respects, participants across activities had 
similar reasons for seeking individual professional improvement: 77 percent of them said that 
they participated for the purpose of “obtaining ideas or contacts that would strengthen my 
work.” 



In general, participation in Consortium professional development was strong. Across 
all the surveyed activities, 86 percent of respondents said they attended “everything.” Single- 
event activities characteristically drew high rates of attendance. Participation was more 
sporadic in the longer-term, school-based activities — the Models Learning Network, where 6 
of 1 1 respondents attended all meetings, and at the Action Research sites, where 10 of 16 did 
so. SCIMAST’s Customized Training — which, like Action Research and the Models 
Learning Network, provided a series of professional development experiences over a long 
term — also experienced fluctuations in attendance. For example, at one site where evaluators 
observed training in the third year of a 3-year program, only about half of the participating 
teachers had also participated in the previous 2 years. Clearly, these findings reflected the 
time constraints that face practicing teachers, whose opportunities to attend professional 
development activities are typically limited. They could also reflect some degree of teacher 
turnover. 

Participants’ reports on the quality and effects of the Consortia’s professional 
development activities inevitably reflected differences in the purposes, expectations, and 
commitment of different groups of respondents. The next section presents participants’ 
reports on the quality of the professional development activities. 
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Participants’ Ratings of Quality and Utility 

Participants gave high marks to the quality of professional development offerings. 
More than 80 percent of respondents surveyed rated the Consortia’s work as good or 
excellent in every respect. Ratings of excellent predominated with respect to accuracy and 
presentation (Exhibit 7). 



Exhibit 7 

PARTICIPANTS’ RATINGS OF THE QUALITY OF PROFESSIONAL DEVELOPMENT 



Rating 


Percentage of Participants 


Accuracy of Content 


Presentation 


Completeness 


Depth 


Excellent 


60 


56 


43 


34 


Good 


39 


32 


44 


50 


Fair 


1 


12 


9 


14 


Poor 


0 


0 


4 


3 



Note: Columns may not sum to 100 percent because of rounding. 

Exhibit reads: Sixty percent of the participants in professional development activities rated the 
accuracy of content as excellent. 

Source: Customer survey. 

Breaking down the results by activity showed some differences in the overall quality 
that participants ascribed to them. TAAMSS rated especially high on all the scales. Training 
of Trainers stood out in one positive respect and in one negative: it received a high rating for 
accuracy (69 percent excellent) but a comparatively low one for presentation (35 percent 
excellent). In interviews, several participants criticized the presentation skills of some 
members of the SCIMAST staff. Nonetheless, they gave the overall activity very high 
marks. 

The fact that participants gave comparatively low ratings to the quality of the Models 
Learning Network activity (relative to the very high ratings participants awarded other 
activities) may have reflected both the small number of participants (around 20) and the fact 
that the activity had only just started. It may also have reflected teachers’ unfamiliarity and 
perhaps dissatisfaction with an activity in which they were expected to lead rather than more 
passively absorb presentations from outside experts. The Consortium staff member 
commented that the group initially asked for more outside help, but then learned to take more 
responsibility for leading the discussions. These survey results suggest that at least some 
participants may have remained uncomfortable with a more active role and continued to feel 
that something was missing. A clearer initial presentation of purposes and roles would have 
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helped, as the Consortium staff have recognized as a result of their early experience with this 
kind of work. 

Survey respondents also reported that the activities in which they took part were closely 
connected to broader reform efforts in mathematics and science education: 76 percent 
characterized the connection as “close.” The most notable exception was among teachers 
who participated in Action Research, who tended to think of “broader reform” as something 
that did not include them: 7 of 15 respondents chose the response “I don’t know; I am not 
involved in broader reform efforts.” 

Across all surveyed activities, most participants (57 percent) considered their activity “a 
good start” toward achieving their purposes, and 36 percent said the activity had served their 
purposes “completely.” Interestingly, participants who came to the activity with the clearest 
purpose in mind — those who were seeking help with a planned organizational 
improvement — were somewhat more likely to have had their purposes completely served; 49 
percent of them chose this response. Next, the chapter turns to a discussion of the effects of 
the professional development activities on participants’ skills and knowledge. 

Effects on Participants’ Skills and Knowledge 

On average, participating in a Consortium professional development activity 
contributed in a “moderate” or “major” way to most participants’ knowledge and skills in 
each of several domains (Exhibit 8). The extent of the contribution varied somewhat by the 
domain of knowledge. Participants reported the largest contribution to their knowledge and 
skills in professional development (where 86 percent called the contribution major or 
moderate). 

In addition to these aggregate findings across all professional development activities, 
the evaluation team also reviewed responses for each activity. The activities that had the 
strongest focus on professional development were TAAMSS and Training of Trainers; 67 
percent and 50 percent of participants, respectively, reported that these activities had made a 
major contribution to their knowledge and skills in professional development. These two 
activities had higher-than-average proportions of staff developers participating, and the 
preexisting interests of these participants probably explain some of their higher ratings. 
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Exhibit 8 



PARTICIPANTS’ REPORTS ON THE EFFECTS OF CONSORTIUM PROFESSIONAL 
DEVELOPMENT ON THEIR SKILLS AND KNOWLEDGE 



Contribution 


Math/Science/ 

Technology 

Education 


Professional 
' Development 


Assessment/ 

Standard 

Setting 


Planning 


Collaboration/ 

Communication 


Major 


24 


28 


22 


19 


26 


Moderate 


52 


58 


42 


44 


40 


Small 


21 


10 


30 


29 


29 


None 


3 


4 


6 


9 


6 



Note: Columns may not sum to 100 percent because of rounding. 

Exhibit reads: Twenty-four percent of the participants in professional development activities reported 
that the activity made a major contribution to their knowledge and skills in math/science/technology 
education. 

Source: Customer survey. 

Some quotations from journals and feedback letters sent by Training of Trainers 
participants give a sense of the kinds of knowledge about professional development that 
participants gained: 

I will plan to include this activity in the professional development provided to teachers 
in our project. 

I learned more strategies for feedback that we will use. 

Similarly, the evaluation data that SERVE collected on TAAMSS included many positive 
comments on this subject, such as this one: 

The Academy offered great insight into issues regarding professional development and 
reinforced views of working in the area of professional development. 

This evaluation’s findings about the contribution of Consortia activities to participants’ 
knowledge and skills in professional development have an important bearing on the 
implementation of one part of the program’s statutory charge: the law suggests that Consortia 
train classroom teachers to provide training to other educators. To do so effectively, one 
would expect that the Consortia should impart skills in professional development to 
participating teachers. This evaluation had a chance to look at the reported effects of a large 
effort of this kind, the Consortium at AEL’s Regional Training Project. The purpose of this 
activity was to prepare participants, who were mostly teachers, to offer professional 
development to others. According to participants, however, the content of the training may 
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have fallen short of what they needed: only 18 percent credited it with contributing to their 
knowledge and skills in professional development in any major way. 

This problem suggests that if Consortia faithfully carry out this legislative mandate, 
they should intensify their efforts to train teachers to be effective staff developers. Besides 
the Consortium at AEL’s Regional Training Project, the SERVE Consortium’s Project Soar 
faces a challenge in, as one district official described it, “being totally immersed in the 
concept of teachers as staff developers.” Through the leadership development activities that 
have begun recently, the SERVE Consortium and the district should focus on developing 
teachers’ staff development skills. 

One activity stood out for its high level of attention to the domain of planning. 

Training of Trainers made a major contribution to the planning skills of 52 percent of its 
participants. The written comments that the SCIMAST Consortium collected from 
participants suggested that requiring school staff to attend Training of Trainers in teams was 
effective: 

Also extremely beneficial was the inclusion of the collaborators in the training session 
along with the school staff. This gave us an opportunity to discuss the training and its 
application to our project. We did so much planning after each daily session. 

Finally, there was considerable variation in the extent to which participants learned 
about communication or collaboration. This variation reflected the diverse purposes of the 
activities. At the high end, 7 of the 1 1 participants in Models Learning Network (64 percent) 
reported that the activity had made a major contribution to their skills in communication and 
collaboration, 52 percent reported the same for TAAMSS, and 42 percent for Training of 
Trainers. The Consortium staff member working with Models Learning Network described 
the strides that school-based participants had made in communicating over the course of the 
project up to then: 

We saw a lot of change in the group. Each time we met, we revisited the purpose. 
Initially, the teachers treated it as a task force or a committee and asked what they were 
supposed to do. During this early period they often asked for outside help or resources. 
As we continued to meet, the group became more independent and interdependent. 

They came to rely on each other as experts. They tended to place much higher value on 
what each other brought to the activity than on things they could get from outside. 

Action Research participants who were interviewed commented very favorably about 
the value of teacher discussions in this activity: 

The meetings gave us an excuse to talk about math content, which we otherwise 
wouldn’t discuss. 
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We would never have had cross sharing before, because there was never a place to 
share. 

This is not a plan from on high. You have to do it, to make your own decisions. 

Instead of feeling powerless by people telling us what to do, this made us feel powerful. 

Two interviewees revealed that TAAMSS’ focus on communication was particularly 
valuable. First, TAAMSS emphasized communication and collaboration as an essential 
foundation of the change process: 

The main point was: in order to make change take place, you must have everyone 
involved; stakeholders are integral to all change. We worked on how to involve them 
and move them along in the process. 

Second, TAAMSS enabled participants to communicate with skilled colleagues from other 
organizations: 

There were very powerful, intellectual, and motivated people from different districts. 
You could see they could make changes in math and science. They were just regular 
teachers. I was in awe about what they could offer. 

Despite a few specific areas where survey findings pointed to weaknesses, participants 
responded very positively when asked about the knowledge and skills they had gained. 
Professional development activities outstripped other kinds of Consortium activities (teams, 
networks, and dissemination of a book of promising practices) in their contribution to 
participants’ knowledge and skills (as later chapters of this report will describe). This 
chapter turns next to the payoff from this skill building: the behavioral change that could be 
attributed to participating in Consortia professional development activities. 

What Participants Have Done as a Result of Professional Development 

Although this evaluation could not directly assess the extent to which participants 
changed their behavior as a result of their involvement with the Consortia, it did ask. 
participants, through both surveys and interviews, to address changes in their own behavior. 
The evaluation inquired about behavioral change in three areas: individual professional 
practice, communication with others, and organizational policies or practices. Most 
participants reported behavioral change at the individual level, although the change was often 
only a first step. 

Nearly two-thirds — 62 percent — of the individuals who had participated in the 
activities surveyed for this evaluation reported that they had incorporated some new behavior 
into their jobs as a result of what they had learned. There was little variation across activities 
in this regard. Some variation was associated with the reason that participants took part in 



the activity, however. Among those who were looking for help with a major, planned 
improvement effort, 75 percent reported incorporating something they had learned. This 
relatively high level of change is entirely plausible, since these participants were in the best 
position to put their new knowledge to work on the job. 

Many Training of Trainers participants commented (in written feedback collected by 
SCIMAST) both that they had learned new things and that they were applying their 
knowledge: 

I felt that I learned and reaffirmed much about how a learning experience should be 

structured. I have already changed some of the methods I am using. 

I am leaving with more of an understanding about what is expected and what I can do 

to help. It was great! 

Although participants in this and other professional development events testified to 
having made changes in their professional practice, participants in more intensive, sustained 
professional development experiences found out that sometimes the more one tries to change, 
the more one appreciates how much more change is needed. Action Research, for example, 
involved teachers in monthly sessions of 45 minutes (at the Denver site) or 2 to 3 hours (at 
the St. Louis site), plus classroom visits by a Consortium staff member. After what the 
Consortium staff member called the “very experimental” first year of Action Research, a 
participating teacher captured this ambivalence in her comment: 

It was far better than I hoped. I got a lot out of it and the kids really improved. I also 

feel it helped with my instruction, but I have a long way to go. 

Some teachers gave specific examples of how they changed their practices. Making 
greater use of manipulatives and focusing more on problem-solving strategies to help 
students with story problems in mathematics were two examples that came up. However, the 
overall structure of teachers’ lessons did not change dramatically, according to Consortium 
staff and the evaluators’ on-site review of a small number of lesson plans. 

Similarly, when evaluators observed an event at a Customized Training site, the effects 
on classroom practice seemed certain to fall short of expectations because it was clear that 
many participants absorbed only part of the content. In an exercise that was designed to 
teach aligning instruction and assessment, most teachers focused on the practical aspects of 
instruction instead. On the other hand, because this was an especially ambitious change 
effort, one could reasonably assess it against the high standard of transforming instruction. 
Participants, in general, were pleased with the intensity and the results of Customized 
Training and offered evidence to support their praise. For example, a principal whose 
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teachers participated in Customized Training at another site commented that their teaching 
practices changed after they had attended the professional development program: 

We have more use of manipulatives and hands-on. We have a science lab, and I’ve 
seen the use increase this year and a move to actual experimentation rather than just 
demonstration. More teachers are asking their students to write about math solutions — : 
why rather than how. 

The law also directed the Consortia to create ripples of communication and learning 
within the regions. Seventy-five percent of all the participants of Consortia professional 
development events surveyed said they had shared something they learned with one or more 
colleagues. Participants varied little in this response across activities, but a slightly higher 
proportion (89 percent) was found among those participants who were looking for help with a 
planned organizational improvement. 

Evaluators also gathered information on the extent to which a Consortium activity 
stimulated and supported further training in the region. As of late spring 1996, the 
Consortium at AEL tracked its effects most systematically. According to its annual report, 
the 350 participants in that year’s Regional Training had (as of the end of April) conducted 
137 workshops, with another 64 scheduled through September. This was an average of just 
over one workshop conducted or scheduled, as of that early date, for every two original 
participants. Similarly, the SERVE Consortium obtained a commitment from TAAMSS 
participants to provide at least two workshops by the end of the summer. The Consortium 
planned to document participants’ compliance with this commitment. Of the two participants 
interviewed for this evaluation, one had met the commitment in June and one expected to do 
so during the summer. An interviewee commented: 

I made a commitment to be part of education reform and science reform, so I will go 
beyond what SERVE asked me to do, to help whoever needs it. If I can’t do it, I’ll look 
to TAAMSS to find someone who can. 

Compared with the reported effects of Consortia professional development activities on 
individual behavior or interpersonal communication, the reported effects of these activities on 
organizations — schools, school districts, and other agencies in which participants work — were 
somewhat more modest. Survey respondents reported that the most prevalent effect on 
organizations was the most low-key effect: the professional development activity “helped 
generate awareness of new information within the organization.” Across the board, 69 percent 
of respondents said the effect in this regard was either moderate (42 percent) or major (27 
percent). There was not much variation by activity. The most striking effect that these 
activities could have had at the organizational level would have been that they “helped the 
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organization initiate a new program, policy, or practice.” Overall, participants split almost 
evenly among the four possible responses to this statement: 23 percent reported a major effect 
of this kind, 25 percent moderate, 26 percent small, and 26 percent no effect. 

The organizations that received grants under SCIMAST’s Professional Development 
Awards program were an exception to the general pattern of Consortia professional 
development activities producing moderate organizational effects. The effects in these 
organizations were unusually large. Having benefited from both the Professional 
Development Award and the Training of Trainers event, 65 percent of survey respondents in 
Training of Trainers reported that their Consortium activities had major effects on helping 
initiate a new program, policy, or practice. 

It thus appears that there was a real payoff from SCIMAST’s care in screening 
applicants and selecting only those with a demonstrated commitment to change. For 
example, applicants had to show that the majority of teachers at each school were involved 
and that administrators joined teachers in planning and training activities. Several 
participants commented on the rigor of the screening process for applications. Consortium 
staff were concerned, however, about the shortage of applicants with the organizational 
capacity to tackle an ambitious program of instructional improvement. In 1995-96, only 9 
strong 'applications were received for the 15 grant slots; 6 additional applicants received 
planning grants. In interviews conducted in the spring of 1997, Consortium staff reported a 
continuing decline in interest in this kind of professional development activity. 

Some Consortia activities that directly served classrooms also affected the surrounding 
organization. Again, an example comes from SCIMAST, which provided Customized 
Training only after it had obtained the principal’s commitment and promise to participate. A 
Consortium staff member observed: 

We’ve found it’s very important to include the administrators. It’s difficult for a 

teacher to run a noisy, investigative classroom when the principal thinks all classrooms 

should be quiet and students sitting in straight rows. 

Similarly, WERC staff claimed success in developing leadership capacity in Arizona 
Journey Schools, thanks to the active involvement of principals as team members. A SERVE 
Consortium staff member working on Project Soar, which tries to effect change across the 
schools in feeder patterns, said flatly: “You can’t get anything done without principal 
support. [In this case,] principals said they wanted this to happen.” 

Only one of these professional development activities set out to tackle the greater 
challenge of affecting not only organizations but also the relationship between schools and 
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communities. Here, the Consortium staff involved with Arizona Journey Schools reported 
less success. Although the teams were supposed to include community members, creating a 
meaningful role for parents and other people from the community turned out to be beyond the 
reach of even this ambitious, 2-year effort. 

Consortia professional development activities varied in the attention they gave to the 
surrounding state policy system. Sometimes their alignment with policy was overtaken by 
events. This happened in Arizona, where WERC’s leadership role in Arizona Journey 
Schools emerged after a drastic change in the state policy climate around educational reform. 
TAAMSS, on the other hand, engaged a fairly high proportion of people who administer state 
programs right from the start. It had a dual focus on instructional content and the change 
process; its format encouraged active networking; and program administrators called it a 
great success in delivering content in a format that served their purposes. Initially, state 
leadership gave another activity, Participant Support, offered by SMCNWS, more mixed 
reviews. This program, which supported professional development for teachers, was 
designed to serve needs directly expressed by teachers. State officials interviewed in the 
spring of 1996, however, said it was not closely enough aligned with state program priorities. 
However, a year later, most state officials were reportedly pleased with the program and its 
fit with state priorities. 

Participants’ self-reports do not provide the ultimate verdict on the effects of a 
professional development activity, of course. Some Consortia planned to gather and analyze 
data on long-term changes in classroom instruction associated with their most ambitious 
interventions. They are to be commended for this commitment to investigating their own 
effectiveness. Interviews with Consortia staff conducted for the evaluation and a review of 
Consortia annual reports suggest that much of this evaluation is still in the planning stage or, 
if under way, has not yet been completed or the results reported. Nevertheless, the kinds of 
data reported here — on perceived contributions of Consortia professional development 
activities to knowledge, skills, and behavior — provide a useful glimpse of the effects that can 
plausibly be attributed to these Consortia services. 

Summary of Strengths and Weaknesses 

Participants in Consortia professional development activities generally praised the 
quality of the offerings and the contributions they made to their skills, knowledge, and 
changes in behavior. Most participants surveyed (62 percent) said they were doing 
something differently as a result of the professional development they received. Fewer, but 
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still about half, said that their organizations were doing something different. Although 
participants came to the events for many different purposes, most of them found a good fit 
between what they sought and what they took away. 

Responding to the law’s charge to create ripple effects in mathematics and science 
education in their regions — to assist and train selected educators so as to have indirect effects 
on larger numbers — the Consortia adopted training-of-trainers strategies and served selected 
sites more intensively. The findings suggest that in choosing the training-of-trainers strategy, 
it is important for the Consortia to attend to three issues in choosing trainers to train: first, 
they should target participants who already are responsible for providing professional 
development as part of their job; second, if the Consortia are going to rely on people in other 
role groups, such as teachers, they should be sure to provide them with training in the specific 
skills of conducting professional development; and third, they should follow up to remind 
participants of their obligation to deliver training to others (and to measure the extent to 
which this is occurring). Data collection subsequent to the surveys did not examine the 
extent to which the participants actually provided training and technical assistance to others. 
This will be an important topic for future evaluations and reports from the Consortia on the 
performance indicators. 

Participating teachers acknowledge that they still have far to go in changing their 
classroom instruction, suggesting that even more depth and persistence are likely to be 
necessary — and, as also noted in the earlier discussion of work in intensive sites, persistence 
may be a problem when participation fluctuates over time, as it does at some sites. Yet the 
depth of discussion and engagement created at local sites was clear from interviews with 
participants. Perhaps, over time, the study groups created in the Models Learning Network 
and Action Research will pay off in real changes in classroom practice. 

Alternatively, it is possible that SCIMAST’s strategy of very rigorously screening 
participating organizations and working only with those that showed an organizational 
commitment to improvement from the start is a necessary ingredient in intensive site work. 
The early, positive findings from Training of Trainers participants (who are also Professional 
Development Award recipients) and Customized Training participants showed the promise of 
this selective approach. Down the road, this Consortium also intended to measure the 
instructional change associated with these intensive interventions. As of the spring of 1996, 
the only real stumbling block that had emerged was the scarcity of organizations that were 
clearly ready to move forward in instructional improvement. The Consortium was trying to 
develop more capacity through planning grants. 



Thus, the Consortia tended to achieve the most impressive results in professional 
development when they worked with individuals and, especially, with organizations that 
were poised to do something — to deliver services to others or to engage in organizational 
change. There must be a good fit between the activity’s purpose and participants’ 
professional responsibilities, and probably their existing capacities as well. In the continuing 
challenge to make the most of limited dollars, the Consortia should strive to form 
partnerships with participating educators who are in a strong position to follow up on their 
professional development. 
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IV. TEAMS AND NETWORKS: CONSORTIA LINKS TO THE FIELD 



The statute reauthorizing the Regional Consortia Program directs the Consortia to work 
cooperatively with others in their regions who are trying to reform mathematics and science 
education. Specifically, it directs them to: 

• Communicate, on a regular basis, with entities within the region who are delivering 
services to students and teachers of mathematics and science. 

• Assist in the development and evaluation of state and regional plans and activities 
that hold promise of bringing about systemic reform in student performance in 
mathematics and science. [Sec. 13302 (11&12)] 

As part of their efforts to establish and maintain these lines of communication, the 
Consortia convene state teams and networks of professionals from across their regions who 
are active in various capacities in mathematics and science education. According to 
Consortia staff, these teams and networks help connect the Consortia to their regions and 
guide some of their decisions about activities and resource allocations. In the spring of 1996, 
the primary purpose of the state teams was to advise the Consortia on opportunities and 
options for providing services in individual states. Some teams had initiated projects of their 
own in their states. The networks concentrated on substantive themes or tasks, such as the 
development of a new mathematics or science curriculum framework. 

Consortium support was intended to advance work in these areas, usually across the 
service region. Participation on both teams and networks afforded members opportunities to 
connect with and learn from people who shared similar or complementary interests and 
concerns. Participation, particularly on the state teams, could also result in members’ 
gamering resources for their own or other projects back home. Finally, as survey data 
indicate, members of teams and networks reported that these activities sometimes contribute 
to changes in how they do their work. Data collected in the spring of 1997 indicate that the 
teams and networks continue to serve the functions identified earlier. 

Team and network members were based in colleges and universities, state education 
agencies, local school districts, and other organizations. Many were grantees of federally 
funded programs, such as NSF’s SSI Projects and the Eisenhower State Curriculum 
Frameworks Projects. Many, if not all, state teams also included state Eisenhower program 
staff. In addition, at least some teams and networks included individuals from informal 
mathematics and science centers, such as museums and the national energy laboratories, and 





from business and the community. Typically, membership on state teams was broader than 
that on networks, reflecting, at least in part, the different purposes of the two types of groups. 

Supporting these teams and networks has become a significant part of most Consortia’s 
portfolios of activities and service. Six Consortia convened and supported state teams, and 
four convened and supported at least one network as of spring 1996. Almost all of these 
groups were established during the first grant period, although they varied somewhat in their 
maturity and prominence in individual Consortia’s portfolios of service. For example, the 
Mid- Atlantic Eisenhower Consortium for Mathematics and Science Education (Mid- Atlantic 
Consortium) organized its state teams early in the initial grant period, and continues to 
support them. 

Supporting teams and networks also became a major activity of the new Consortium in 
the Northeast and Islands region. The Eisenhower Regional Alliance for Mathematics and 
Science Education Reform (Regional Alliance) continued to support the state teams and two 
regional networks (the Equity Steering Committee and the K-16 Steering Committee) 
originally organized by a different grantee. In addition, this Consortium has organized four 
new networks. One is the Alliance Schools Network, connecting schools from each of the 
nine jurisdictions in the Regional Alliance’s service region. According to the Regional 
Alliance’s 1997 annual report and recent interviews, the Alliance Schools Network has 
become a prominent part of the activities and services of the Regional Alliance. It has also 
become the primary vehicle for delivering this Consortium’s intensive services to 
participating schools in the states and territories that it serves. 

The second new network is the Curriculum, Instruction, and Assessment Network, 
which has the goal of supporting state and local efforts to implement curriculum frameworks. 

In its 1997 annual report, the Regional Alliance also discusses early efforts to convene an 
Informal Science Network. This is an outgrowth of the Consortium’s efforts to identify 
informal science centers for an on-line database operated by the Consortia and the 
Eisenhower National Clearinghouse (ENC), an activity that is discussed in more detail in the 
chapter of this report on collaboration and networking among the Consortia. The fourth new 
network is the Public Engagement Network. 

The state teams supported by three other Consortia — the Consortium at the Appalachian 
Education Laboratory (Consortium at AEL), the High Plains Consortium (HPC), and the 
SERVE Eisenhower Consortium for Mathematics and Science Education (SERVE 
Consortium)— developed later than the teams in the Mid- Atlantic and Northeast regions. For 
example, the SERVE Consortium convened its state teams but, as of spring 1996, they had met 
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only once. The Science and Mathematics Consortium for Northwest Schools (SMCNWS), the 
new Consortium in the Northwest, created state advisory groups, but their work was halted 
until October 1996. Subsequently, these groups resumed their activities and have taken an 
active role in advising the Consortium on policy issues related to its programmatic activities. 
The Southwest Consortium for the Improvement of Mathematics and Science Teaching 
(SCIMAST) established the Southwest SSI Network. This network continues to be the only 
one that the Consortium supports. Similarly, the Mathematics and Science Leadership Team 
was the only network supported by the Pacific Mathematics and Science Regional Consortium 
(Pacific Consortium). Staff of the WestEd Regional Consortium for Science and Mathematics 
(WERC) participated on the California Science Education Advisory committee, which was 
producing a science curriculum framework for the State Superintendent of Public Instruction. 

The Sample of Teams and Networks Evaluated 

The evaluation examined and surveyed a sample of participants from 9 of the 10 teams 
and 4 of the 5 networks. The teams included at least one team from each Consortium that 
supported teams and all the networks that the Consortia supported at the time. To guide the 
selection of sample teams, evaluators invited Consortia directors to nominate teams that they 
believed to be particularly active in the states. Data collection activities included interviews, 
document review, and surveys of team and network members. The sample of team and 
network participants who were surveyed included a total of 3 19 members. Exhibit 9 displays 
the teams and networks included in the survey sample, the numbers of members on the official 
rosters of these groups provided by the Consortia, the sample size, and the survey response 
rates. In addition to the teams and networks surveyed, the evaluators collected some 
information on the state team in the District of Columbia, which the Mid- Atlantic Consortium 
supports, and the Pacific Consortium’s Pacific Regional Mathematics and Science Team. 

Activities of the State Teams 

From the perspective of the Consortia, the primary reason for convening and supporting 
state teams is to connect the Consortia to the field. The Consortia expected state teams to do 
this in several ways: 

• Provide advice about state needs and opportunities for service by the Consortia. 

• Make decisions about allocating resources available from the Consortia. 

• Inform others about the work of the Consortia. 

• Work on activities related to state reform initiatives. 



In addition, the Consortia viewed their support of state teams as creating opportunities for 
people who otherwise might not come together to discuss issues of common concern and to 
learn from each other. 



Exhibit 9 

MATHEMATICS AND SCIENCE EDUCATION REGIONAL CONSORTIA PROGRAM 

PARTICIPANT SURVEY SAMPLE 

Teams 



Team 


Consortium 


Total 

Participants 


Sample 

Size 


Surveys 

Returned 


Response 
Rate (%) 


Tennessee Steering 
Committee 


Consortium 
at AEL 


38 


38 


28 


74 


Vermont Statewide 
Action Team 


Regional 

Alliance 


12 


12 


8 


67 


Maine Statewide 
Action Team 


9 


9 


8 


89 


New Jersey State 
Team 


Mid-Atlantic 


124 


58 


35 


60 


Pennsylvania State 
Team 


70 


28 


23 


82 


Florida State Team 


SERVE 


10 


10 


8 


80 


North Carolina State 
Team 


9 


9 


7 


78 


Alabama State Team 


10 


10 


5 


50 


North Dakota Math 
and Science Alliance 


HPC 


34 


34 


19 


56 


Total 


316 


208 


141 


68 



Networks 



Network 


Consortium 


Total 

Participants 


Sample 

Size 


Surveys 

Returned 


Response 
Rate (%) 


California Science 
Education Advisory 
Committee 


WERC 


32 


32 


23 


72 


K-16 Steering 
Committee 


Regional 

Alliance 


26 


26 


21 


81 


Equity Steering 
Committee 


24 


24 


15 


63 


Southwest SSI 
Network 


SCIMAST 


29 


29 


21 


72 


Total 


111 


111 


80 


72 
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All the teams included in the second phase of the evaluation carried out all these 
functions, but there are differences in how they did so. Three examples illustrate some of the 
differences: 

• Tennessee State Steering Committee. This state team, like the others organized by 
the Consortium at AEL, advised the Consortium director and staff on a number of 
activities that the Consortium intended to carry out in the region. Team members 
reviewed grant proposals for several Consortium-sponsored mini-grant programs 
and selected the projects to receive Consortium support in Tennessee. Team 
members also actively assisted in reframing the content of the Consortium’s 
Regional Training Project (discussed in this report’s chapter on professional 
development) to link it more explicitly to state mathematics and science reform 
efforts. Team members then helped the Consortium identify schools to receive 
intensive assistance. The steering committee also advised the Consortium on the 
content of the Regional Training Project. Like the other state steering committees, 
this one recommended that the content of the Regional Training Project address the 
state curriculum framework for mathematics and science education and that an 
earlier workshop package on national standards be revised. 

• Pennsylvania State Team. With guidance and support from the Mid- Atlantic 
Consortium, this state team identified a set of priorities for contributing to 
mathematics and science education reform in the state. Guided by these priorities, 
the primary function of the team was to select activities that addressed these 
priorities to support with Consortium resources. Proposals for individual projects 
usually came from team members. According to the Consortium’s 1997 annual 
report, this team set three priorities for its upcoming work: (1) supporting 
professional development by creating regional collaboratives; (2) disseminating a 
coherent message about reform to various stakeholders; and (3) promoting equity 
through professional development and dissemination of information about 
encouraging female and minority students in mathematics, science, and technology. 

• North Dakota Math and Science Alliance. Working with Consortium staff, this 
state team assessed needs and identified priorities for activities that were consistent 
with the overall priorities of the Consortium. Using financial and other resources 
from HPC, it convened a number of statewide and regional meetings that focused 
on reforms in mathematics and science education and set priorities for state 
initiatives. A science focus group also evolved out of the activities of this team. 

As the three examples illustrate both the availability of Consortium resources to support 
the teams themselves and the activities that the teams chose were key ingredients in the glue 
that held the teams together. Indeed, all the Consortia that supported state teams invited them 
either to make resource allocation decisions or to advise the Consortia on allocating 
resources. Three Consortia — HPC, SERVE, and the Regional Alliance — allocated equal 
amounts of Consortium funds to each of the teams in their regions to support team activities 
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or projects. The amounts ranged from $10,000 per team at SERVE and the Regional 
Alliance to $35,000 at HPC. The Mid- Atlantic Consortium allocated resources to the four 
teams in its region according to a formula based on enrollments. 

Directors and staff of Consortia that supported state teams generally agreed that state 
teams were important “sources of intelligence about what’s going on in the states” and that 
“they let us know about what the states need.” “They are the central actors in defining needs, 
activities, and priorities of the Consortium.” One director gave a more specific example of 
the kind of strategic information that might come from a state team: 

The main purpose of the teams is to keep us informed about major issues in each 
state — about changes in math and science or in the frameworks and assessments. For 
example, if team members say that they weren’t very involved in the framework writing 
because it was a top-down effort, then we would know how to look at it [the 
framework] when we get to it. 

The Consortia made every effort to keep state teams apprised of Consortia activities 
and services — according not only to interviews with Consortia directors and staff but also to 
a review of minutes and summaries of team meetings and the survey data. In most cases, 
team meetings convened by the Consortia included presentations and reviews of ongoing and 
new Consortium activities and services. In addition, interviews with Consortia staff and 
directors and members of the teams, as well as survey data, indicated that the Consortia 
actively followed up with team members. The Consortia fully expected team members to 
share their information with colleagues in their organizations or with constituent and 
stakeholder agencies and organizations. Through this process, the Consortia hoped that, as 
people in the regions became more aware of their activities and services, they would come to 
see the Consortia as valuable resources. As one Consortium director, who thought that the 
state team strategy had achieved its purpose quite effectively, explained: 

We expect them [state team members] to actively disseminate the information they get 
from us to their organizations, and it appears that they are doing that. In fact, this 
model has worked so well for us that we are replicating it in the comprehensive 
center.... Sometimes the [team members] serve as our best public relations people. 

These communications on behalf of the Consortia also served another function, at least 
by implication. If team members successfully communicated messages about the resources 
that were available from the Consortia, they helped limit expectations and led clients and 
potential clients to make requests that were consistent with what the Consortia could do. 
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Activities of the Networks 



The following brief profiles describe the activities of the networks prior to the 
administration of the customer survey in spring 1996: 

• Equity Steering Committee (Regional Alliance). The purpose of this network, 
which includes state agency and local staff responsible for addressing equity issues 
in mathematics and science education reform, was to create a regional 
infrastructure. The Regional Alliance sponsored annual institutes that focused on 
specific topics (e.g., data use) related to solving local equity problems. The 
Consortium also provided resources for members to attend other training and , 
professional development sessions. Network members advised the Consortium on 
other activities to support. Activities during 1996-97 continued to focus on data use 
in monitoring equity problems. 

• K-16 Steering Committee (Regional Alliance). This network brought together K-16 
staff and faculty and staff from institutions of higher education who were interested 
in K-16 collaboration as a strategy in systemic reform. Annual institutes, which 
were sponsored by the Regional Alliance, included opportunities for network 
members and other participants to visit partnerships of K-12 and higher education 
institutions. The primary activity of this network in 1996-97 was a regional 
conference on professional development schools. 

• Southwest SSI Network (SCIMAST). Members included SSI directors and staff 
who met to explore issues of common concern related to state systemic reform. 
Meetings generally focused on single topics related to the work of all the SSI 
projects, such as assessment, curriculum frameworks, and standards. Directors 
made up the core group, and individual project staff participated in activities related 
to their specific responsibilities. The Consortium provided financial support for this 
network and assumed responsibility for organizing and facilitating network 
meetings. 

• California Science Education Advisory Committee (CSEAC) at WERC. With the 
endorsement of the state superintendent of public instruction, this group of 
California science educators, which was convened with the California Alliance for 
Science and Math, the state’s SSI project, worked on the development of standards 
for science education in the state’s Challenge District Program. Consortium staff 
participated as members of the network and provided technical as well as financial 
support for this network. The Consortium’s Web site was used to disseminate 
drafts of the standards and to collect comments. 

• Pacific Regional Mathematics and Science Team (Pacific Consortium). Network 
members included mathematics and science supervisors from each of the 10 entities 
in the Pacific region. With financial and technical support and training from the 
Consortium, members of this network drafted mathematics and science standards 
for the Pacific region and worked on implementing them. Planning was under way 
for network activities and projects to be completed during the current grant period. 



Networks differed in their clarity and specificity of purpose, and their activities 
reflected these differences. Two networks, the Southwest SSI Network and the CSEAC, had 
reasonably clear — albeit quite different — purposes. The purpose at the Southwest SSI 
Network was to support the ongoing implementation of SSI projects, whereas the CSEAC 
took on the challenging and politically charged task of writing science standards. The 
Regional Alliance assumed the ambitious but less well-defined tasks of working with the 
Equity and K- 1 6 Steering Committees to define key issues in each of these areas and to 
organize activities that would help participants understand and address them. 

Consortia support for network activities ranged from organizing routine meetings to 
supporting special projects. In addition, for four of the five networks, Consortia assumed a 
major leadership role in convening and facilitating meetings. In the case of the CSEAC, 
Consortium staff played a membership rather than a leadership role. Finally, like the state 
teams, the networks were all sources of ideas and advice about additional activities the 
Consortia could support. Interviews with Consortium staff suggested that the Equity 
Steering Committee, the K-16 Steering Committee, and the Pacific Regional Mathematics 
and Science Team played an especially valuable role in this regard. 

State Policy Context of the Teams and Networks 

State context, as defined by the presence or absence of ongoing reform initiatives in 
mathematics and science education and the status of the state department of education, 
appeared to exert a significant influence on the work of the state teams. In the spring of 
1996, observations about the Consortia experiences in organizing and convening the state 
teams included in this evaluation led the evaluation team to conclude that it was easier to 
work with teams in places where there was less state-level action in mathematics and science 
education reform. For example, in reviewing the differences among the four states served by 
the Consortium at AEL and the progress made in working with the teams, the Consortium 
director observed: 

The Consortium is not the only game in town, particularly in Kentucky and Virginia. 

We are the only game in town in Tennessee. Therefore, it has been easier for us to 
work with the Tennessee Steering Committee, and this committee has made the most 
progress.... [In contrast] all of the changes in Virginia have made it harder for us to 
work there and even to know who we are to work with. 

Similarly, staff at the Mid-Atlantic Consortium and HPC attributed the slow progress of 
the Pennsylvania State Team and the North Dakota Math and Science Alliance, respectively, 
to the relatively limited role that the two state departments of education played in 
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mathematics and science education. In Pennsylvania, there were no major mathematics and 
science education reform initiatives under way. As for North Dakota, a member of the North 
Dakota Math and Science Alliance said: 

HPC has been the main funding source for the Math/Science Alliance, and this funding 
has allowed our alliance to move from its beginnings as a small advisory group which 
nearly died [because of a reduction in the state department of education] to an action 
group which has greatly enlarged and is taking a leadership role in promoting 
mathematics and science in this state. 

In states with major reform initiatives, teams and networks needed to link their work to 
these initiatives in some way. The experiences of the CSEAC and the New Jersey State 
Team represented contrasting examples of efforts to find a niche in existing state reform 
activities. The CSEAC’ s development of standards for science education was intended to 
contribute directly to the state’s Challenge Schools initiative, and the state superintendent of 
instruction endorsed this activity. Consortium support helped make the work possible. In 
New Jersey, the state team was organized as part of the New Jersey SSI project. Staff at the 
Mid-Atlantic Consortium pointed out that since elements of the state’s mathematics and 
science reform were organized under the New Jersey SSI project, which was based at Rutgers 
University, it did not make sense to create a separate entity or one that was not closely linked 
to the SSI. A member of the team explained the reasons for what he saw as the team’s lack 
of impact: 

In New Jersey, the state team has no role other than to figure out how to spend the 
funds allocated by the Mid- Atlantic Consortium. As a team, it’s not very proactive... it 
doesn’t have a role for two reasons. First, we have an SSI here. That should be taking 
into consideration all of the needs in math/science/technology education in the state.... 

In a sense, the SSI functions like a state team.... Second, we have a strong mathematics 
coalition... which functions like a state team in math issues. The Math Coalition did this 
prior to the SSI and serves as a model. So you have two strong organizations, each 
having a similar role, overlapping already themselves... the state team does not have a 
* major effect. 

The experiences of two Consortia, the Regional Alliance and the Consortium at AEL, 
suggested that major changes in state departments of education seriously hindered the work 
of state teams, at least in the short term. Changes in Virginia’s state department of education, 
including a change in leadership and a highly charged statewide debate over standards, made 
it difficult for the Consortium to even know whom they should contact. Similarly, the co- 
directors at the Regional Alliance argued that the size and complexity of the New York State 
Department of Education, combined with recent changes in leadership and structure, made it 
difficult for the Consortium to organize a team in that state or even to find opportunities for 
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service. One co-director also noted that “recent downsizing of state departments [in the 
Northeast] has caused a lot of stress, and people are very worried about their future. It’s 
difficult for us to get teams to focus for these reasons.” Subsequently, the Regional 
Alliance’s 1997 annual report indicates that the Consortium has made progress in reshaping a 
plan for a state team in New York. 

Participants and Participation in Teams and Networks 

Members of teams and networks reflected a range of experience and perspectives on 
mathematics and science education and education reform. Surveys provided information 
about three characteristics of these individuals that were likely to influence the internal 
dynamics of teams and networks, as well as the benefits that were likely to accrue to the 
participants and to the Consortia: where the members worked, their job responsibilities, and 
their reasons for participating on a team or network. 

Composition of the Teams and Networks 

Typically, individuals were invited to join teams and networks because they were in 
key positions and were expected to know about and to have an interest in the work of the 
Consortia. This strategy was consistent with the Consortia’s primary purpose in supporting 
the teams and, to a lesser degree, the purpose of the networks, which was to advise the 
Consortia on activities and investments. One Consortium director explained how his 
Consortium convened the state teams in its region: 

We started with the categories of people in the legislation. We got representatives from 
each group. To start, we wanted the key stakeholders, such as the state Eisenhower 
people, K-12 people, postsecondary coordinators, state supervisors of math and science, 
key representatives of professional associations, key business people, and district-level 
math and science people. The teams are intended to be inclusive.... We have an ad for 
members in our brochure and in [the Consortium’s newsletter]. People can fill out an 
interest form and become members. 

Just under a third of the team and network members who responded to the survey 
reported that they worked in institutions of higher education (Exhibit 10). About 28 percent 
of the respondents worked in schools or school districts, and about 1 1 percent worked in state 
education agencies. A substantial number also reported working in some other kind of 
organization. This general pattern held for both teams and networks. 
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Exhibit 10 

MEMBERSHIP ON TEAMS AND NETWORKS, BY LOCATION OF EMPLOYMENT 

(In Percentages) 



Employment Location 


All Participants 


Teams 


Networks 


School 


11 


14 


3 


School district 


17 


17 


17 


Intermediate education agency 


6 


6 


3 


State education agency 


11 


10 


15 


Institution of higher education 


31 


34 


25 


Other 


24 


19 


37 



Exhibit reads: Eleven percent of the members of teams and networks who responded to the 
survey reported that they were employed in a school. 

Source: Customer survey. 



When asked about their job responsibilities, 73 percent of the members of teams and 
networks reported that professional development was a main part of their responsibilities 
(Exhibit 1 1). Overall, teams included more staff from schools and more elementary and 
secondary school teachers than did networks. 

The survey data indicate that the Consortia achieved broad-based membership for the 
teams and networks. At the same time, most members of teams and networks were not 
formal representatives of the organizations in which they worked or to which they belonged. 
This absence of formal ties could weaken the Consortia’s connections to the field. The 
selection and formal charge to members of the state teams in the Consortium at AEL’s region 
are an example of a strategy to overcome this problem. The Consortium director reported 
that “all members of the steering committee are designated and recognized as representing 
their respective organizations and associations.” Thus, the chief state school officers were 
involved in selecting state agency staff to serve on the state team. For representation from 
state professional associations (e.g., the state mathematics or science teachers association), 
the Consortium invited the president, or a designee, to join the team. The Consortium 
director thought that this process “creates stronger links to these organizations.” 
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Exhibit 11 



MAJOR PROFESSIONAL RESPONSIBILITIES OF THE 
MEMBERS OF TEAMS AND NETWORKS 
(In Percentages) 



Responsibility 


All Participants 


Teams 


Networks 


K-12 teaching 


18 


22 


8 


Postsecondary teaching 


26 


28 


20 


Administering local education 
program(s) or organizations 


26 


26 


28 


Administering state education 
program(s) or organizations 


25 


23 


29 


Providing professional 
development or technical 
assistance 


73 


69 


82 


Other 


16 


18 


15 



Note: Columns may not sum to 100 percent because multiple responses were possible. 

Exhibit reads: Eighteen percent of the members of teams and networks who responded to 
the survey reported that they were teaching in grades K-12. 

Source: Customer survey. 

Reasons for Participating on a Team or Network 

Overall, teams and network members who responded to the surveys had similar 
purposes for participating (Exhibit 12). The most frequently mentioned reason was “being a 
resource to others”; 77 percent of the respondents listed this reason. Just under two-thirds of 
all participants reported that they were seeking to keep current professionally or that they 
were looking for ideas or contacts that would strengthen their own work or strengthen the 
work of their organization. Fifty-three percent were looking for ideas and contacts to help 
with a planned change effort. 

Reasons for participating on teams and networks differed among role groups. When 
compared with state agency staff and faculty and staff from institutions of higher education, 
school and school district personnel more often said that keeping current professionally and 
gaining ideas and contacts to improve their own work were reasons for joining one of these 
groups. On the other hand, state agency staff and higher education faculty and staff 
mentioned serving as a resource to others more often than did school and district personnel. 
Overall, the survey data suggest that local personnel were more often motivated by individual 
interests and concerns to participate on teams and networks. 



Exhibit 12 



MEMBERS’ PURPOSES FOR PARTICIPATING IN TEAMS AND NETWORKS 

(In Percentages) 



Purpose for 
Participation 


All 

Participants 


Location of Employment 


School 


School 

District 


Intermediate 

Education 

Agency 


State 

Department 
of Education 


Institution 
of Higher 
Education 


Other 


Keeping current 
professionally 


63 


69 


75 


72 


57 


58 


62 


Obtaining ideas or 
contacts that 
would strengthen 
my work 


67 


74 


70 


72 


64 


60 


73 


Obtaining ideas or 
contacts that 
would strengthen 
my organization’s 
existing work 


66 


50 


55 


80 


84 


72 


62 


Obtaining ideas or 
contacts that 
would help my 
organization carry 
out a major, 
planned 

improvement effort 


53 


43 


61 


64 


60 


55 


44 


Being a resource 
to others 


77 


64 


66 


80 


76 


91 


5 



Note: Columns may not sum to 100 percent because multiple responses were possible. 

Exhibit reads: Sixty-three percent of members of teams and networks who were employed in schools 
reported that one reason for participating was to keep current professionally. 

Source: Customer survey. 



One member of a state team very succinctly stated a reason for participating on a state 
team: “I expected that being on [the state team] would affect my organization, that we would 
get some funds for the projects we do.” The prospect of obtaining resources from their 
Consortium motivated others to serve on teams and networks as well, according to interviews 
with members and some Consortia staff. Because one of the functions of the state teams and 
some of the networks was to decide how Consortium resources were allocated and there was 
a possibility that at least some of those resources might support activities for which group 
members had responsibility, it was not surprising that they would be interested and willing to 
serve. 
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Level of Involvement 

Among all the survey respondents who reported belonging to a team or network, 

55 percent described their participation as “interested and involved” (Exhibit 13). Slightly 
fewer than a fifth, 1 8 percent, said that they were “critical” to the work of the team or 
network. At the other end of the spectrum, 17 percent indicated that they were “available if 
they need me,” and 1 1 percent said that they were “uninvolved.” 



Exhibit 13 

LEVELS OF INVOLVEMENT OF MEMBERS OF TEAMS AND NETWORKS 

(In Percentages) 



Self-Reported 
Level of 
Involvement 


All 

Partici- 

pants 


Location of Employment 


School 


School 

District 


Intermediate 

Education 

Agency 


State 

Department 
of Education 


Institution 
of Higher 
Education 


Other 


Critical to the 
activity’s success 


18 


8 


22 


0 


12 


17 


25 


Interested and 
involved 


55 


51 


31 


52 


56 


64 


61 


Available if 
needed 


17 


30 


24 


8 


20 


15 


9 


Uninvolved 


11 


12 


24 


40 


12 


4 


5 



Note: Columns may not sum to 100 percent because of rounding. 

Exhibit reads: Eighteen percent of the members of teams and networks reported that they were critical 
to the activity’s success. 

Source: Customer survey. 

Role groups varied in their levels of involvement in teams and networks. Eighty-one 
percent of higher education faculty and staff and 68 percent of state education agency staff 
said that they were either interested and involved in a team or network or critical to its 
success. Conversely, 59 percent of school staff and 53 percent of school district staff said 
that they were interested and involved or critical to the success of a team or network. 

Members of state teams were somewhat less likely than network participants to report 
being interested and involved or being critical to the teams’ success (69 percent versus 78 
percent). Seventy-five percent or more of the members of six teams and three networks 
reported that they were either interested and involved or critical to the teams’ success. In 
contrast, a third or more of the members of three state teams — Florida, New Jersey, and 
Pennsylvania — said that they were either available if needed or uninvolved. (Eighteen 
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individuals identified as members of the New Jersey state team to whom we sent surveys 
reported that they either were not on the team or were unsure whether they were.) 



Challenges Posed by Diverse Participants and State Context 

Supporting teams and networks placed Consortia in relatively long-term relationships 
with the groups and individual members. These relationships represented substantial 
investments for the Consortia, both in the funds they provided for team and network activities 
and in the staff time and energy they devoted to convening and facilitating these groups. The 
similarities and differences in the professional experience and perspectives of the individual 
participants, combined with the influence of state policy context, posed some real challenges 
to the Consortia. They had to somehow find ways to bring all these people together to have 
productive conversations and to work on common tasks. They also had to find ways of 
working within vastly different state contexts. In general, these challenges were greater 
among the state teams and greater for some teams than others. Finally, the fact that more 
faculty and staff from institutions of higher education and from state departments of 
education than local educators reported being more involved and active than local educators 
raises the possibility that local voices were less often heard. 

Consortium directors and staff who worked with the teams and networks were aware of 
these challenges. One, who said that he was both surprised and frustrated by first contacts 
with state teams, offered the following description of his Consortium’s early struggles: 

Everybody had an agenda. It took me a year and a half to get everyone to look at the 
big picture and to come to a common vision. When we first came into the picture, 
everyone thought that we were a source of money and that’s what everyone wanted 
from us. Now all of the [state teams] are pretty much moving forward in the same 
direction. 

A staff member at the Mid- Atlantic Consortium added: 

It’s difficult to get different people to agree on common goals. Working with and 
through a lot of different people means that it is time-consuming to develop genuinely 
collaborative relationships. The different teams have different successes and failures. 
Our frustration is how to get their work coordinated in a focused way.... The closest the 
teams have come is through the support of professional development. [Professional 
development] has gotten people around the table to think of the big picture. 



Impact and Outcomes of the Teams and Networks 



Members of the teams and networks reported that their activities created new 
opportunities for communication to occur and laid the groundwork for new working 
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relationships. Members also reported that their participation affected their knowledge and 
skills and how they did their work, although these effects appeared to be fairly modest for 
most participants. At the same time, it appeared that participants from institutions of higher 
education and state departments of education derived more from their participation than did 
either school staff or school district personnel. 



Effects on Individual Behavior and Skills 

Participants in the survey sample reported that being on teams and networks had a 
variety of effects on their work. Almost three-quarters of them said that they had shared 
ideas from these activities with their colleagues. In addition, just under 60 percent said that 
these groups confirmed things that they were already doing. At the same time, 54 percent 
also reported that they had incorporated something new in their work. More than half of the 
members of six teams, but only one member of the networks, reported incorporating 
something new in their work as a result of participating in these groups. 

Members gave mixed ratings to the contributions of teams and networks to their 
knowledge and skills in (1) mathematics/science/technology education, (2) professional 
development, (3) assessment/standard setting, (4) planning, and (5) collaboration and 
communication. These ratings appear to be consistent with their overall assessment of how 
s well the teams or networks served their purposes. Contributions in the area of collaboration 
and communication received the highest rating, with 44 percent saying they were major and 
32 percent saying they were moderate. Ratings in the other four categories tended to cluster 
in the moderate and small range, although between 15 percent and 22 percent rated the 
contributions in these areas as major. 

Participants also reported some effects of the teams and networks on their 
organizations. Forty-five percent said that the addition of new knowledge was a moderate 
effect and 40 percent that confirming what their organizations were already doing was a 
moderate effect. Team members were more likely to report these effects than were members 
of networks, who more often saw these as small effects. Just over a fourth of the members of 
the teams and networks saw the teams and networks as having moderate effects on initiating 
something new in their organizations, and more than half said that the groups had no effect 
on efforts to do something. 
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Extent to Which Teams and Networks Served Participants’ Purposes 

Overall, 58 percent of the participants said that the team or network was a good start 
toward serving their purposes in participating, and 28 percent said that their purposes were 
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completely served. Ten percent said that the team or network was a start but that it did not 
provide sufficient information or guidance for follow-up. At least 70 percent of the members 
of all the teams and networks reported either that their purposes were completely served or 
that the team or network represented a good start (Exhibit 14). 



Exhibit 14 

MEMBERS’ REPORTS ON THE EXTENT TO WHICH 
TEAMS AND NETWORKS SERVED THEIR PURPOSES 
(In Percentages) 





All 

Partici- 

pants 


Location of Employment 


Extent to Which 
Teams/Networks 
Served Members’ 
Purposes 


School 


School 

District 


Intermediate 

Education 

Agency 


State 

Department 
of Education 


Institution 
of Higher 
Education 


Other 


Purposes 
completely served 


28 


35 


11 


55 


20 


32 


32 


Good start 


58 


35 


77 


46 


65 


56 


59 


A start, but not 
sufficient informa- 
tion or guidance 
for follow-up 


10 


8 


10 


0 


16 


9 


7 


Did not serve 
purposes 


4 


22 


3 


0 


0 


3 


2 



Note: Columns may not sum to 100 percent because of rounding. 

Exhibit reads: Twenty-eight percent of the members of teams and networks reported that their 
purposes in participating were completely served. 

Source: Customer survey. 



Among the four role groups discussed above, faculty and staff from institutions of 
higher education reported that the team or network was either a good start (56 percent) or 
served their purposes completely (32 percent). More local school district staff (77 percent) 
and staff from state education agencies (65 percent) reported that the team or network was a 
good start, but only 1 1 percent of the former and 20 percent of the latter reported that their 
purposes were completely served. Thirty-five percent of school staff said that their purposes 
were fully served, although 30 percent said either that their purposes were not served (22 
percent) or that the team or network was a start but did not provide sufficient information or 
guidance for follow-up (8 percent). 
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Outcomes of the Work of Teams and Networks 

Activities sponsored or carried out by state teams tended to fall into two categories: 
conducting or supporting meetings on issues related to mathematics and science education 
reform, and carrying out short-term workshop or training activities. Often, Consortia 
resources represented only a portion of the resources necessary for the activities, but they 
leveraged other resources. Compared with Consortia support for state teams, Consortia 
support for networks represented longer-term and more focused engagement on a particular 
task or in a particular area. Thus, the early work of the Pacific Mathematics and Science 
Team resulted in the development of mathematics and science standards for the entities in the 
Pacific region. Similarly, there was the possibility that the ongoing work of the CSEAC 
would produce new science education standards for California. The Southwest SSI Network 
provided ongoing support to SSI project staff and facilitated sharing among the projects. 
Finally, support for the Equity Steering Committee and the K-16 Steering Committee 
represented ongoing efforts to define issues in these two areas of mathematics and science 
reform. 

When asked to identify significant outcomes of the work of the teams and networks, 

89 percent of the members mentioned opening important new lines of communication. 

Slightly fewer, 77 percent, agreed that these groups provided unique forums for discussing 
important issues and created productive new working relationships. Seventy-seven percent 
also identified support for worthwhile projects related to reforms in mathematics and science 
education as a significant contribution. Members of the teams and the networks differed 
considerably in their assessments of this contribution. Eighty-five percent of the team 
members said that this was an important contribution, perhaps because their projects back 
home were among those that received support. In contrast, only 58 percent of the network 
members saw support of worthwhile projects as an important contribution. 

Despite members’ reports that significant outcomes of the teams and networks included 
support of worthwhile reform-related projects, there has been little systematic review of the 
quality and impact of the projects supported by the teams and networks. For example, the 
director of a Consortium in which state teams sponsor professional development said simply: 
“We have not followed up on whether they have implemented the training.” Another director 
said: “We know what they do, but we don’t know very much at all about whether any of it is 
any good.” 
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Participant Ratings of Consortia Support for State Teams and Regional 
Networks 

This section discusses participants’ ratings of the support that the Consortia provided to 
teams and networks. Members of all but two of these groups gave the Consortia solid marks 
for their support. Overall, approximately 60 percent of team and network members said that 
the Consortia’s paying for the groups’ activities, convening and facilitating their meetings, 
and contributing staff time to organizing the teams and networks had major effects on these 
groups. Providing financial and other support for participants and providing substantive 
advice and guidance were rated as having major effects by 45 percent and 42 percent of ail of 
the participants, respectively. Producing or disseminating a team or network product was 
rated as a major effect by 34 percent of the participants. The fact that the teams and networks 
are not, for the most part, involved in product development explains this last finding. 

Members of the teams and networks valued different kinds of support. For example, 

73 percent of the members of the networks rated facilitation of the networks as having a 
major effect, whereas only 53 percent of the members of the teams rated this contribution as 
having a major effect. Among all the networks, this kind of support was most often rated as 
having a major effect. Members of state teams were more likely to rate Consortia financial 
support for team activities as a major contribution. Indeed, members of all but one state 
team most often cited financial support for team activities as having a major effect. 

Finally, members of state teams were more likely to see the Consortia’s substantive 
advice and guidance as having a major impact than were members of the networks (46 percent 
versus 32 percent). This finding may be explained by the fact that the membership of state 
teams was more diverse than that of the networks, so that additional external guidance and 
direction were required to pull the groups together. The more clearly defined substantive focus 
and tasks of the networks may also mean that, once they were organized, they required less 
external guidance and direction. 

Another dimension of Consortium support for teams and networks was follow-up. 

Survey data indicate that the Consortia were very aggressive in several kinds of follow-up. 

For example, 82 percent of all team and network members said that the Consortia had sent 
them materials more than once. Seventy-five percent said that follow-up had included more 
than one workshop, training, or meeting, and 72 percent said that it had included more than 
one telephone call. Fifty-five percent said that follow-up included providing financial 
support more than once. 
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Survey findings about the leadership of the teams and networks were mixed. Overall, 

5 1 percent of the participants said that leadership was equally balanced between the 
participants and Consortia staff. At the same time, 20 percent said that it came from 
Consortia staff, and 18 percent said that it came from the participants. Participants on four 
teams and all the networks were most likely to see leadership as a shared responsibility. 
Participants on two state teams, those in Maine and Vermont, saw the teams themselves as 
exerting leadership. Participants on the Alabama state team saw Consortium staff as exerting 
leadership. Overall, the findings about team and network leadership suggest that there was 
some sharing of responsibility for this function and that participants were assuming 
ownership of the groups. 

Overall, the evaluation data on participants’ ratings of Consortia support for teams and 
networks suggest that these groups looked to the Consortia for financial support and for 
support in arranging and facilitating group activities, presumably including follow-up with 
materials and communication. They were somewhat less likely to look to the Consortia for 
leadership and substantive advice and guidance, although for some members of teams and 
networks, and for the teams and networks as groups, these were not unimportant 
contributions. 

Strengths and Weaknesses of Consortia Support for Teams and Networks 

Members of teams and networks valued participation in these groups for opportunities 
to open lines of communication with people whom they otherwise might not have met, to 
discuss important issues, and to forge new working relationships within their states and 
across regions. A large majority also said that these activities were closely related to broader 
reform efforts. Significant numbers of the participants, particularly those on the state teams, 
said that these groups supported important contributions to reform in mathematics and 
science education. Somewhat fewer said that their participation influenced their own work or 
that of their organization in some way. 

The findings about members’ involvement in teams and networks suggest that 
individuals from the local level — schools and school districts — were somewhat less involved 
in these groups than individuals from institutions of higher education and state departments 
of education. There are a number of possible explanations for this pattern, including the 
difficulties that local staff may face in getting away from work and traveling to a meeting. 
Strategies such as the recent reconfiguration of the Pennsylvania state team into seven 
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smaller regional entities are promising examples of efforts to make these groups more 
accessible to members and to ensure that all of their voices are heard. 

The presence or absence of state reform initiatives and the degree of stability in state 
departments of education defined a Consortium’s structure for working with and through a 
state team. In states where there were limited mathematics and science education reform 
initiatives, Consortium support could at least partially fill a void, and the state team could 
become a catalyst for reform. The North Dakota Math and Science Alliance and the Pacific 
Regional Mathematics and Science Team are examples of how this process worked. In states 
where there were more ambitious reform efforts, the Consortia and the teams had to find a 
niche and were more likely to devote considerable time and energy to negotiating a 
productive role. Finally, transitions in the leadership and organization of state departments of 
education could mean that decisions about team membership and team activities had to be 
put on hold until new players were in place and new agency priorities had been set. 

Are the interests of the Consortia well served by their support of the teams and 
networks? The evaluation findings indicate that these activities did help define the Consortia 
as conveners and facilitators of potentially important dialogues, as catalysts for new working 
relationships, and as providers of financial resources. Team and network members viewed 
them as sources of leadership and guidance somewhat less often. Working with and through 
the teams and networks appeared to connect the Consortia to the field. The Consortia still 
faced the challenge of ensuring that these connections were strong and that they reflected 
good strategic choices for the use of Consortia resources in facilitating, supporting, and 
assessing the work of the teams and networks. In addition to looking at the contributions to 
mathematics and science education reform achieved through support for the teams and 

networks, the evaluators also inquired about the extent to which sponsoring these groups 

\ . 

served the interests of the Consortia beyond meeting the objective of supporting reform. 
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V. PROMISING PRACTICES 



The Consortia’s authorizing legislation charges them with “disseminating 
exemplary mathematics and science education instructional materials” [Sec. 13301(a)(1)], 
Therefore, conducting initiatives related to exemplary or promising practices is a logical 
extension of their purpose. The identification and dissemination of instructional 
materials, teaching methods, or assessment tools that are somehow determined to be of 
high quality fell within the legislative charge, as did providing technical assistance to 
sites that were implementing the promising practices. 

The identification and dissemination of promising practices was a comparatively 
low-profile activity for the Consortia during 1995-96. Consortia annual reports for 1996- 
97 indicate that some Consortia are expanding their efforts in both print and electronic 
media, although these still do not appear to represent a major investment for them. The 
evaluators examined this topic during the spring of 1996 in part because the Office of 
Educational Research and Improvement (OERI) was launching a new approach to 
identifying promising practices but had not yet developed a plan for disseminating the 
practices. 2 Because evaluation data documenting the Consortia’s experiences in this area 
could help inform emerging policy and programs, the focus of this section of the report is 
on the lessons the Consortia learned about the process and payoff of identifying and 
disseminating promising practices. Accordingly, this chapter analyzes the “market” for 
promising practices by analyzing recipients’ responses to a book of promising practices in 
mathematics and science education. 

Activity Evaluated 

As part of their effort to respond to the charge to disseminate mathematics and 
science education instructional materials, the Consortia jointly developed Promising 
Practices in Mathematics and Science Education and published it in 1995. This book 
provided 50 two-page descriptions of programs and practices (along with shorter 
summaries of another 67 programs identified in an earlier, related effort by the regional 
educational laboratories). This was the only recent work on promising practices that was 
reasonably uniform across several Consortia and for which recipients could be 
systematically surveyed for this evaluation. (Much of the Consortia’s activities with 

2 Subsequently, in the summer of 1997, OERI’s Mathematics and Science Education Expert Panel 
completed a set of standards and guidelines for identifying promising and exemplary practices. 

According to OERI staff, two Consortia, WERC and the Mid-Atlantic Consortium, were assisting in 
pilot testing the standards and guidelines. 



promising practices involves electronic dissemination — but, as discussed elsewhere in 
this report, the actual recipients of electronic communication are hard to track with 
current technology.) By the spring of 1997, the Consortia had updated much of the 
original material in the book and, in collaboration with the Eisenhower National 
Clearinghouse (ENC), put it into an electronic database. 

The publication of this book was the culmination of a nationwide search and 
screening of promising practices. Some Consortia vigorously pursued practices that would 
be suitable for dissemination. They publicized the effort extensively and worked hard to 
arrive at operational criteria reflecting innovativeness, support of national standards, 
effectiveness, and transferability. They also coped with local sites’ reluctance to cooperate 
in a labor-intensive identification process. A Consortium staff member describes the 
shortcomings of the process that ultimately resulted, as well the division of labor: 

Our stuff was not validated. When we went to the schools, [school staff] said that 
they couldn’t or wouldn’t fill out the forms.... We needed people to fill out the 
forms, but they wouldn’t do it. There was no incentive.... Later, we agreed to 
streamline the form. We reduced it to 15 pages. Then we decided that the 
Consortium could fill out the form. 

Beginning in October 1995, several Consortia disseminated the book. The history of 
this effort mirrors in important ways the policy that seemed to be emerging from OERI in 
the area of promising practices, that is, (1) the process of identifying the book’s contents 
was very elaborate and labor intensive, and (2) dissemination reflected a low-level 
investment and a passive approach, consisting, in this case, of mailings rather than in- 
person communication or assistance. 

The Survey Sample and Recipients of the Book 

Five Consortia that disseminated copies of the volume made mailing lists that they 
could retrieve and make available to the evaluators. The 10 Consortia also disseminated 
thousands of copies but did not retain records of the recipients. The mailing lists 
contained the names and addresses of 1,444 recipients. Findings from our survey of 
recipients from these lists should be interpreted with caution because we do not know 
how representative they are of the total population of book recipients. Moreover, more 
than 50 percent of our final sample comes from one Consortium. See Exhibit 15 for 
details on the survey sample. 3 



3 The Midwest Consortium for Mathematics and Science Education (Midwest Consortium) had records of 
two quite different mailings, one to people who were themselves potential disseminators of the book and 
one to the Consortium’s much larger, general-purpose regional mailing list. 



Exhibit 15 



RECIPIENT SURVEY SAMPLE: DISSEMINATION OF PROMISING PRACTICES IN 
MATHEMATICS AND SCIENCE EDUCATION— 1995 



Consortium 


Total 

Recipients 


Sample Size 


Surveys 

Returned 


Response 
Rate (%) 


Consortium at AEL 


77 


77 


51 


66 


Mid-Atlantic 


34 


34 


17 


50 


SCIMAST 


56 


56 


32 


57 


Midwest 


37 


37 


21 


57 




1,226 


245 


133 


54 


HPC 


14 


14 


7 


50 


Total 


1,444 


463 


261 


56 



According to the Consortia, survey respondents were individuals who either 
requested the volume or, more commonly, received it because their names were on a 
mailing list. The Consortia did not conduct any personal follow-up to underscore the 
book’s message (and they had not been charged with doing so). However, recipients who 
were already part of the Consortia’s work, such as the members of state teams in the 
region of the Consortium at AEL, continued their contact with the Consortia. 

Most individuals among the 56 percent who responded to the survey were teachers 
and/or providers of professional development or technical assistance (Exhibit 16); 43 percent 
of respondents fell into each of these categories. Local administrators were also well 
represented, at 28 percent. Thus, the book had reached local educators. 

Exhibit 16 

PROFESSIONAL RESPONSIBILITIES OF INDIVIDUALS ON MAILING LISTS 



Responsibility 


Percentage of Respondents 


K-12 teaching 


43 


Postsecondary teaching 


12 


Administering of local education program(s) or organization(s) 


28 


Administering of state education program(s) or 
organization(s) 


13 


Professional development or technical assistance 


43 


Other 


7 



Note: Column does not sum to 100 percent because multiple responses were possible. 

Exhibit reads: Forty-three percent of the individuals who responded to the survey reported that 
they taught at the K-12 level. 

Source: Customer survey. 
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The Consortia acknowledged that disseminating a book to a mailing list was a low- 
intensity activity from the recipients’ perspective. Not everyone who was surveyed 
remembered receiving it. Of those who responded to the survey, 62 percent remembered 
receiving the volume. It is likely that many of those who did not respond — and 
nonrespondents were especially numerous for this activity — did not recall receiving it, but, 
of course, the evaluators had no firsthand information to confirm this suspicion. Of the 
nearly two-thirds of respondents who did say they received the book, half said they had 
read the entire volume or most of it; half had read less than half, very little, or nothing. 

When asked about their purposes in looking at the book, recipients (defined here as 
those who remembered receiving it) tended to report either a personal or interpersonal 
purpose (Exhibit 17). About three- fourths said their purpose was “generally keeping 
current professionally”; about two-thirds expected to strengthen their own work; the same 
proportion intended to use it in helping others. Somewhat fewer expected their reading of 
the book to have organizational effects. 

The survey also revealed that the people on these mailing lists tended to be in closer 
touch with their Consortia than most educators in their regions. For example, 34 percent 
had received telephone calls from their Consortia, and 23 percent reported that they had 
attended workshops, training sessions, or meetings sponsored by their Consortia — clearly 
a higher proportion than would be found among educators generally. Thus, the responses 
reported here are probably somewhat more favorable than would be obtained if the book 
were a completely stand-alone intervention. For a number of respondents, the book was 
part of a larger series of interventions. 

Exhibit 17 

RECIPIENTS’ PURPOSES IN LOOKING AT THE BOOK 



Purpose 


Percentage of Recipients 


Keeping current professionally 


74 


Obtaining ideas or contacts that would strengthen my work 


66 


Obtaining ideas or contacts that would strengthen my 
organization’s existing work 


53 


Obtaining ideas or contacts that would help my organization 
carry out a major, planned improvement effort 


39 


Being a resource to others 


66 



Note: Column does not sum to 100 percent because multiple responses were possible. 

Exhibit reads: Seventy-four percent of the recipients of the book reported that one purpose in 
looking at it was to keep current professionally. 

Source: Customer survey. 
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Recipients’ Ratings of Product Quality and Utility 



Recipients rated the quality of the book as high on several dimensions (Exhibit 18). 
Ratings for accuracy and presentation were especially high, with about two-thirds calling 
it “good” in these respects and about one-fourth calling it “excellent.” Ratings on 
completeness and depth were only slightly lower. 



Exhibit 18 

RECIPIENTS’ RATINGS OF THE QUALITY OF THE BOOK 



Rating 


Percentage of Recipients 


Accuracy of 
Content 


Presentation 


Completeness 


Depth 


Excellent 


25 


26 


18 


14 


Good 


67 


62 


62 


66 


Fair 


8 


12 


21 


18 


Poor 


0 


1 


0 


2 



The book did not appear to duplicate other available resources; a large majority of 
recipients (79 percent) judged it to be “unique.” A smaller majority (58 percent) called 
it “closely related to broader reform efforts” in mathematics, science, and technology; 

29 percent said it was “somewhat related” to such efforts; 13 percent responded “I don’t 
know; I am not involved in broader reform efforts.” 

Effects on Recipients and Their Organizations 
Effects on Skills and Knowledge 

About three-fifths of survey respondents credited the book with making either a 
moderate or a major contribution to their skills and knowledge related to mathematics, 
science, or technology education (Exhibit 19). About half said it had made a moderate or 
major contribution to their skills and knowledge related to professional development. As 
would be expected for a resource book, respondents who called its contribution 
“moderate” far outnumbered those who called it “major.” Its contributions in the areas of 
assessment or standard setting, planning, and collaboration or communication were 
reportedly smaller, with about one-fifth of respondents saying it had made no 
contribution to their skills and knowledge in these areas. 




61 



75 



What Recipients Have Done as a Result of Reading the Book 

Although many recipients considered the book a worthwhile resource for building 
their professional know-how, fewer said they were now doing something differently as a 
result of reading it. Fifteen percent of recipients agreed with the statement “I have 
incorporated something I learned from this activity into my job”; this is a far lower 
percentage than was found among participants in professional development, state teams, 
or networks who had responded to the same statement. Just 1 percent said the book had a 
major effect on their organization’s initiation of a new program, policy, or practice, 
although 28 percent did attribute a moderate effect to the book in this regard. 

Exhibit 19 

RECIPIENTS’ RATINGS OF THE BOOK’S CONTRIBUTION 
TO THEIR KNOWLEDGE AND SKILLS, BY DOMAIN 
(In Percentages) 



Contribution 


Math/Science/ 

Technology 

Education 


Professional 

Development 


Assessment/ 

Standard 

Setting 


Planning 


Collaboration/ 

Communication 


Major 


8 


9 


2 


5 


4 


Moderate 


53 


40 


38 


40 


40 


Small 


27 


43 


38 


33 


35 


None 


12 


7 


22 


21 


21 



Note: Columns may not sum to 100 percent because of rounding. 

Exhibit reads: Eight percent of the recipients of the book reported that the book made a major 
contribution to their knowledge and skills in math/science/technology education. 

Source: Customer survey. 

These responses were consistent with the observation of a Consortium director: 

I think that resources may be a barrier to using these practices. People read about 
them and then go to their principal to discuss trying them. The principal says that 
there is no money and that’s the end of it. 

Another measure of the book’s effectiveness was the extent to which recipients 
either passed the information along or collected more information. Thirty-one percent 
shared some of the book’s contents with a colleague. Even more — 41 percent — 
contacted at least one of the sites whose practices were written up in the book. Of those 
who reported contacting sites, 59 percent said they received “a variety of useful 
information,” although 1 8 percent found the information “not very useful,” and 6 percent 
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found “there was little or no additional information available.” In general, then, the book 
had a discernible effect in stimulating communication among educators. 

Summary of Strengths and Weaknesses 

The Consortia’s expectations in producing and disseminating the book were entirely 
consistent with the findings reported here. They believed the book addressed people’s 
interest in receiving information, but, in the absence of more resources to support 
implementation, they expected only a few recipients to take action on the ideas. A 
Consortium director and a staff member summarized the expectations in the following 
ways: 

The book gives general ideas about programmatic approaches recipients could take. 
It gives them contacts to make about a particular program. People who pick it up 
and read it have an interest in improving practices. The ideas they get, they may 
incorporate by virtue of having read the book and thought about it.... It’s great 
materials and information and resources, but any kind of dissemination in broad- 
scale media has a low level of intensity for those who receive it. Some’pick it up 
and it has no impact. 

[Its purpose is] providing easy-to-use information for teachers, professional 
developers, and others who are looking for new ideas. The guides would be 
resource books to help people who were looking for more material and teaching 
practices. 

Given the Consortia’s modest aspirations for the book, recipients reported a 
reasonably high level of satisfaction with the extent to which it served their purposes: 

66 percent called it “a good start,” and 16 percent said “it served my purposes 
completely.” 

One can assess the effectiveness of this work on promising practices from several 
perspectives: (1) the payoff from disseminating the volume, (2) this same payoff in 
relation to all the work on the volume, (3) OERI’s policy choices in setting up a 
framework for grantees’ or contractors’ work in selecting and promoting the use of 
promising practices. 

Viewed solely as a dissemination effort, the promising-practices volume can be 
considered a success. Once the book was compiled, disseminating it was a low-cost way 
of giving educators a high-quality resource. Recipients found the book reasonably useful 
for their purposes, although the purposes were most often limited to gathering 
information or simply staying up-to-date professionally. 
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However, the Consortia used considerable resources to identify candidate programs 
and practices, establish criteria for including them in the volume, review the candidate 
programs, and develop the volume. These were time-consuming procedures. 

Considering the opportunity cost of this work — the other ventures in professional 
development or technical assistance that were not launched — the evaluators have serious 
doubts about the cost-effectiveness of the overall effort. The survey data showed that 
most recipients used the book in a fairly casual way, browsing for ideas and perhaps 
finding some that they passed on to others or (more rarely) put into practice themselves. 
To the evaluators, there seemed to be a mismatch between the elaborate professional 
attention given to identifying practices and the informal, limited attention that readers 
gave to the resulting volume. 

To inform policy decisions about future directions in the spread of promising 
practices, one can look at other avenues that OERI and its grantees or contractors could 
take. One choice would be to accompany an intensive process of selecting practices with 
equally intensive attention to the eventual use of the practices — to provide some technical 
or financial assistance for implementing them, for example. A Consortium director 
suggested this approach, criticizing the existing strategy of developing and disseminating 
materials: 

We don’t have a lot of faith in that strategy. You are telling people, “Here’s a 
practice that works in one place,” but you’re not giving them resources for 
implementation. 

One solution to the problem of the need for follow-up that recognizes the limits of 
the Consortia’s resources was found by the Consortium at SERVE. The Consortium 
conducted a professional development event — Celebrating Success, discussed earlier in 
the chapter on professional development — that focused on promising practices. The 
survey respondents who participated in that event reported considerably greater effects on 
their knowledge and work than did the readers of the book whose responses are reported 
in this chapter. Fifty-nine percent reported that they had incorporated something they 
learned into their work; 50 percent reported a moderate or major effect on a new 
organizational program or policy. These responses support the argument that a modest 
investment in follow-up makes for a much more effective intervention, and that the 
ultimate effects of Consortia efforts to produce products, such as the book on promising 
practices, will be modest and perhaps disappointing with such an investment. 
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VI. THE USE OF TECHNOLOGY BY THE CONSORTIA 



The original authorizing legislation charges that the Consortia “, . .if feasible, 
maintain on-line computer networks with all Regional Consortiums and the National 
Clearinghouse for Science and Mathematics Education Materials....” According to the 
terms of their grants, each Consortium is also required to operate a demonstration site 
within its region featuring services offered by the Eisenhower National Clearinghouse 
(ENC). By the spring of 1996, the Consortia had gone beyond this mandate, using 
technology in many ways to further the goals of the program. 

All Consortia established World Wide Web sites. These sites varied in complexity 
and sophistication, but they all described Consortia activities and disseminated resources. 
Consortia also encouraged educators to use technology by offering them technical 
assistance and professional development. In keeping with the grant award, each 
Consortium maintained an ENC demonstration site. Consortia also used Internet 
technology, through list servers, to promote communication and cooperation among 
various groups that are interested in the reform of mathematics and science education. 

By the spring of 1996, technology represented a fairly small part of the Consortia’s 
portfolio of activities, but there was a clear trend toward increasing the integration and 
use of technology in every Consortium. By the spring of 1997, nearly all Consortia had 
expanded both their Web sites and their use of technology. 

This section describes the ways in which the Consortia used technology, beginning 
with a discussion of the Consortia’s Web sites. Next, it discusses Consortia efforts to 
promote educators’ use of technology through professional development and technical 
assistance. Then it describes the Consortia’s efforts to promote communication and 
networking among state teams, regional groups, and the Consortia network through 
technology. It concludes with a discussion of the strengths and weaknesses of the 
Consortia’s use of technology. 

Consortia on the World Wide Web 

The Consortia and the ENC have a presence on the World Wide Web (see 
Exhibit 20). Using a “browser” (such as Netscape Navigator) and an appropriate Internet 
connection, an individual could locate the Web site of a Consortium. As an example, the 
Midwest Mathematics and Science Consortium (Midwest Consortium) home page was 
located by typing in the address “http://www.ncrel.org/msc/msc.htm” (see Exhibit 20). 
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The local computer retrieved the information that is displayed in Exhibit 21 from a 
computer, or server, located at the Midwest Consortium in Oak Brook, Illinois. 

Exhibit 20 

CONSORTIA ON THE WORLD WIDE WEB 



Eisenhower Regional Math/Science 
Consortium at AEL 


http://www.ael.org/eisen/ 


WestEd Eisenhower Regional Consortium for 
Science and Mathematics 


http://www.wested.org/werc/ 


High Plains Consortium for Mathematics and 
Science at Mid-continent Regional 
Educational Laboratory (McREL) 


http://www.mcrel.org/hpc/ 


Midwest Consortium for Mathematics and 
Science Education at North Central Regional 
Educational Laboratory (NCREL) 


http://www.ncrel.org/msc/msc.htm 


Eisenhower Regional Alliance for 
Mathematics and Science Education Reform 
at TERC 


http://ra.terc.edu/alliance/HubHome.html 


Science and Mathematics Consortium for 
Northwest Schools (SMCNWS) at Columbia 
Education Center 


http://www.col-ed.org/smcnws/ 


Pacific Mathematics and Science Regional 
Consortium at Pacific Resources for 
Education and Learning (PREL) 


http://prel-oahu-1.prel.hawaii.edu/math- 

science/index.html 


Mid-Atlantic Eisenhower Consortium for 
Mathematics and Science Education at 
Research for Better Schools (RBS) 


http://www.rbs.org/eisenhower/ 


Eisenhower Consortium for Mathematics and 
Science Education at SouthEastern Regional 
Vision for Education (SERVE) 


http://www.serve.org/Eisenhower/ 


Eisenhower Southwest Consortium for the 
Improvement of Mathematics and Science 
Teaching (SCIMAST) at Southwest 
Educational Development Laboratory (SEDL) 


http://www.sedl.org/sedl/scimast.html 



This exhibit displays one of the primary features of the World Wide Web, the use of 
links, shown as underlined text. By using a mouse to click on School Improvement 
“Pathways,” for example, a user would immediately move to the Pathways Web page 
(see Exhibit 22). On this Web page, there is much more that the reader could access. 

This includes information on “Parent and Family Involvement,” “Assessment,” “School 
to Work,” “Math,” “Science,” etc., indicated by the underlined words at the bottom of the 
page. A click on any of these takes the user to yet another page with additional 
information on that topic and additional links to related information. 





Exhibit 21 



MIDWEST CONSORTIUM FOR MATHEMATICS AND SCIENCE EDUCATION 

(WEB PAGE) 



yv\v\v. nc- ►*c! . o nq 



MSC 

Features 



Exemplary 

Units 



First in the 
World 



Mission Statement 



Integrated 

Math/Science 

Resources 



Intensive sites 



Math 

Resources 

Science 

Resources 

Status Report 



T he Midwest Mathematics and Science Consortium (MSC) is engaged in providing 
direct technical assistance to state education agencies, intermediate state educational 
units, and local school districts. Activities reflect a commitment to learning based on 
a constructivist paradigm based on best practices and research to optimize the 
opportunities for all students to learn. The result of this technical assistance is to 
advance systemic reform of Mathematics and Science Education providing 
meaningful, purposeful, engaging learning in these areas available to all students. 
Whenever and wherever possible, the Consortium advocates and promotes the use of 
appropriate technology to support and accelerate systemic reform. 




Internet 
Lesson Plan 
Index 



Advisory 

Board 



State Team 



Staff 



1 he MSC provides three Internet sites as electronic support tools. Pcuhwgys to 
School Improvements, which was co-developed with NCREL, features 
research-based information on school improvement with special sections pertaining 
to math, science, and professional development. Learning Through Technology: A 
Planning and Implementation Guide, which was co-developed with NCREL, ISBE, 
and NCSA, is designed to help educators and community members develop a 
comprehensive learning and technology plan. In addition, the Systemic Mathematics 
and Science web site describes innovative approaches to teaching and learning in 
math and science. 



ENC 



Other 

Regional 

Consortia 



Send any questions, comments, and suggestions to Gil Valdez. 



Exhibit 22 

PATHWAYS TO SCHOOL IMPROVEMENT WEB PAGE 





(animated logo in Netscape 2.0+) 

A product of the North Central Regional Educational Laboratory 
in cooperation with the Regional Educational Laboratory network 

Last update: November 19, 1997 

Topics Currently Available: 

( Click on Topics button above for a variety of 
additional resources and a list of topics under development ) 

Assessment | At-Risk | Early Childhood | Governance | Leadership | Learning | Math | 
Parent and Family Involvement | Preservice Education | Professional Development | 
Safe and Drug-Free | School-to-Work | Science | Technology 



Pathways to School Improvement Internet Server Team 



content & general comments : niroi7i)crclorg ; 



■technical information nautaV/mcrcl.org 



All of these features are supported by the World Wide Web, but the Midwest 
Consortium had to design them into its Web page. Such a sophisticated Web site requires 
careful thought and a significant investment of resources. Several of the Consortia have 
placed a priority on being on the cutting edge of technology and the Web. The others 
have established a simpler Web presence. 

In the spring of 1996, the evaluation team completed a review of the Consortia’s 
Web sites. During the summer of 1997, information on the Consortia’s Web sites was 
updated. In both reviews, the Web sites of the 10 Consortia varied in size, sophistication, 
and content. The evaluators conducted a detailed examination of three Consortia Web 
sites that had an extensive Web presence in 1996 and updated that review in 1997. 

In 1996, the WestEd Regional Consortium for Science and Mathematics (WERC) 
had a home page with 22 links, including a link for each of the four states in its service 
area. It referenced a major WERC project (Science Education Academy of the Bay Area, 
or SEABA), listed WERC’s major partners (The California Alliance for Mathematics and 
Science and the University of California, Irvine), and included six topic areas: standards- 
based reform, content and pedagogy, assessment, equity in mathematics and science 
education, technology, and environmental education. The state pages, in turn, identified 
local science and math programs and resources. The topical pages listed relevant services 
and resources available at WERC and the WestEd Lab. At the time of this study, WERC 
was updating its site to reflect changes at its parent organization. It had also recently 
posted an electronic version of a new WERC publication, Tales from the Electronic 
Frontier, in which nine K-12 teachers described how they had used the Internet to 
improve math and science education. In addition, WERC was in the process of setting up 
a list server (a way of communicating through the Internet) to work with a group of 
collaborators on the California Science Standards. The Consortium also planned to 
incorporate the use of on-line interactive databases that would allow various participating 
organizations to contribute current information to the WERC Web site, such as adding 
local activities on a regional calendar of math and science events. 

The WERC Web site in 1997 retained its basic structure but was far more 
sophisticated and offered access to more resources. WERC successfully implemented 
on-line interactive databases to produce a regional calendar of events and to facilitate the 
development and review of science standards for California. WERC leaders cite their 
investments in technology and the expansion of their Web site as particularly successful 
at furthering their overall work. 
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In 1996, the Web site of the Midwest Consortium at the North Central Regional 
Educational Laboratory (NCREL) had 17 links to national organizations (e.g., the other 
Consortia, ENC, AskERIC) and its own efforts and projects. The Midwest Consortium 
benefited from the fact that technology was the special focus assigned to NCREL and that 
NCREL operated the North Central Regional Technology in Education Consortium 
(NCRTEC). In addition to “Pathways to School Improvement” and “Learning Through 
Technology,” the Midwest Consortium Web site referenced more than 60 other 
mathematics and science sites around the country. There was also a Gopher menu with 
hundreds of lesson plans in math and science. 

By the summer of 1997, the Midwest Consortium’s Web site was more fully 
developed. The Midwest Consortium continued to share technology expertise and 
resources with NCREL and NCRTEC. Although “Systemic Mathematics and Science” 
was still under construction, the Consortium’s Web site included more links to what the 
Consortium had identified as exemplary materials, lesson plans, and other resources. 

In 1996, the Hub, the site for the Regional Alliance for Mathematics and Science 
Education Reform, offered users an extensive set of resources on-line. The Regional 
Alliance benefited from the fact that it was located at TERC and was able to take 
advantage of the significant technological experience and expertise available there. The 
Regional Alliance Web site was also very regionally oriented. The site referenced the 
states in its service area through the Alliance Schools Network that was under 
development. It provided a page for each of seven states, Puerto Rico, and the Virgin 
Islands for the 33 schools that were members of the Alliance Schools Network. Each 
state and entity had a page for the statewide action teams that were identifying local and 
state priorities and resources in science, math, and technology. These pages listed links 
to science and math resources in the state. The Regional Alliance Web site provided 
access to numerous mailing lists, or list servers, that allowed users to get and send e-mail 
messages on a range of topics that included assessment, curriculum frameworks, equity, 
evaluation, professional development, and technology. As with other Consortia, the 
Regional Alliance Web site has matured and expanded since 1996. The site contains 
hundreds of links to mathematics and science resources and other organizations. The 
Alliance Schools Network is in place, although the individual school pages are 
underdeveloped. 

In the spring of 1996, five of the Consortia had a very small Web presence, 
including two with a single page listing seven links, most of which were “under 
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construction.” Three of those five had brief home pages that most often described the 
Consortium and its staff and listed six to nine links, which typically included references 
to the National Eisenhower Program, the ENC, and the Department of Education. 
However, by the summer of 1997 all the Consortia had greatly expanded their presence 
on the World Wide Web. For example, the Science and Mathematics Consortium for 
Northwest Schools (SMCNWS) had devoted significant staff resources to the 
development of its well-designed and resource-rich Web site. 

By linking their sites to the ENC, the Consortia were able to meet an important 
responsibility: directing people to the resources of the ENC. The ENC Web site offered 
an extensive listing of resources and services in mathematics and science education, 
along with a collection of on-line documents, publications, lessons, and activities. 
Specifically, the ENC site included a catalog of science and mathematics curriculum 
materials, including print, video, audio, graphic images, software, and kits, from a variety 
of sources. Users could access these materials through the Resource Finder, a search tool 
that located materials on the basis of standard library information, such as author, title, 
subject, and date, along with information more specific to the needs of educators, such as 
grade level, target audience, evaluation, and the relationship of the materials to various 
curriculum standards. The catalog was available on-line via the Internet and on CD- 
ROM. The ENC Web site included a monthly “Digital Dozen,” ENC-selected Web sites 
containing some of the best resources in mathematics and science. ENC Web site users 
could access selected journal articles on mathematics and science education, as well as 
ENC publications. The ENC Web site allowed users to access significant reform 
documents, such as the NCTM standards and the National Science Education Standards, 
as well as a limited number of state curriculum frameworks documents. 

Promoting the Use of Technology through the ENC Demonstration Sites, 
Publications, Professional Development, and Technical Assistance 

All the Consortia promoted the use of technology. Each had established ENC 
demonstration sites as another way to encourage educators to use the extensive resources 
of the ENC. Most Consortia provided direct professional development to teachers in the 
use of technology, usually in the form of summer institutes or short workshops designed 
to introduce teachers to the Internet. In addition, some Consortia lent their expertise to 
states and districts as they developed their technology infrastructures. 
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Demonstration Sites 

The ENC demonstration sites were established primarily for classroom teachers, 
most often those in the immediate geographic area because of the constraints of time and 
travel. In addition, demonstration sites served technology and content specialists. These 
demonstration sites had two stated purposes. One was to give teachers the use of 
workstations and resource materials that they could use to develop lessons. The second 
was to provide teachers and other users with professional development experiences on the 
use of the Internet. 

The sites were sometimes located in the facilities of collaborating organizations and 
sometimes in Consortia offices. For example, the Regional Alliance’s demonstration site 
was at the Boston Computer Society, the Midwest Consortium’s was at the Lederman 
Science Center at the Fermi National Accelerator Laboratory, and WERC’s site was 
located in the Biodiversity Resource Center at the California Academy of Sciences. 

There were several advantages to locating the demonstration sites in these facilities. 

First, these sites were frequently more accessible to the general public and to teachers 
than Consortia offices. In addition, collaborating organizations frequently supplemented 
the resources of the Eisenhower demonstration site program. For example, the WERC 
demonstration site was located just inside the front entrance of the Academy of Sciences, 
a very popular natural science museum in San Francisco’s Golden Gate Park. The two 
computers supplied by the Eisenhower program were augmented by several other, more 
powerful computers owned by the Biodiversity Center. The Center’s extensive collection 
of software and other science-related publications supplemented the software provided by 
the program. Similarly, an additional 40 software programs and 160 CD-ROMs in the 
Biodiversity Center’s collection supplemented the 15 software programs and 30 CD- 
ROMs supplied by the ENC. The potential downside to this arrangement was that the 
Consortia and ENC presence may not have been apparent in the midst of an often much 
larger collection and set of activities in the associated facility. 




By the summer of 1997, several Consortia had changed the locations of their 
demonstration sites. In at least three cases, the Consortia had taken steps to increase 
access to the ENC in their region. The SMCNWS had moved its demonstration site to 
the popular Oregon Museum of Science and Industry and identified 22 colleges, 
intermediate education service agencies, and informal science organizations that agreed 
to make the ENC available at their sites. These ENC Access Centers dramatically 
increased opportunities for teachers in this large region to visit a site where they could be 
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introduced to the ENC resources. In another example of increasing access, WERC 
established a portable network that allows multiple users to access the World Wide Web 
at conferences, meetings, and other sites. Meanwhile, AEL has moved its demonstration 
site to different locations around the region. In addition, AEL and the Regional Alliance 
were working to establish ENC Access Centers throughout their regions. 

Publications 

All Consortia tried to promote the use of technology in mathematics and science 
classrooms by identifying and disseminating useful publications and curricular resources 
on the topic. Some Consortia also developed new publications to address the needs of 
teachers and others interested in expanding the use of technology in classrooms, as in 
WERC’s Tales from the Electronic Frontier. The Mid- Atlantic Consortium’s booklet on 
the Internet for teachers, Internet Jones, had been distributed to more than 26,000 
teachers by the spring of 1997. 

Professional Development and Technical Assistance 

Some Consortia provided teachers with professional development to increase their 
use of technology. Although many Consortia professional development activities 
included technology components, some Consortia sponsored or provided workshops and 
institutes focused on technology specifically. For example, the Consortium at AEL 
collaborated with the Center for Excellence in Mathematics and Science Education at the 
University of Tennessee, Martin, to train teams of higher education faculty and K-12 
teachers in technology. The Midwest Consortium presented workshops to several Illinois 
and Indiana school districts on using technology to support engaged learning. SMCNWS 
helped organize and sponsor a summer institute developed by staff from the Northwest 
Regional Consortium for Technology in Education and the ENC. 

Other Consortia planned to expand their professional development activities on the 
use of technology. For example, both the Regional Alliance and WERC purchased 
portable local area networks (LANs) — sets of portable computers with connections to 
each other and the Internet — to use for off-site workshops on Internet-based resources in 
math and science. As reported in the chapter on networking and coordination, the 
Midwest Consortium was leading a collaborative venture to design and produce a 
professional development toolkit on CD-ROM, now scheduled for completion by 
December 1997. 
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A few Consortia provided technical assistance to state departments of education, 
statewide advisory committees, and individual school districts on the use of technology. 
For example, Midwest Consortium staff provided technology planning assistance to 
school districts and provided technical assistance to an advisory committee for 
technology in Illinois. WERC assisted Nevada in establishing a statewide computer 
network for educators. 

Networking through List Servers 

A list server (or “listserv”) is a way of communicating through the Internet. It is 
similar to e-mail in that one person sends information to another, specified by a particular 
e-mail address. However, with a listserv, the sender can build a list of people who may 
want to communicate regularly for a particular reason and give this list a single e-mail 
address. The benefit is that a person can send one message to everyone on the list by 
using this single address. Any of the people receiving the list can reply, in turn, using 
that same address. Alternatively, all the original messages and their replies can go to a 
listserv manager, who decides which ones to post to the list. 

By the spring of 1996, the Regional Alliance had established more than 70 listservs 
for its statewide action teams, its advisory board and steering committees, and other 
groups. In addition, it provided listservs on a number of topics, including curriculum, 
instruction, assessment, local framework implementation, equity, and K-16 collaboration. 
People interested in these topics subscribed to a list and periodically received information 
from the person who created the list and/or from any of the other members of the list. 

The level of activity on these listservs varied. The equity listserv had 235 messages 
posted from October 1995 through March 1996, whereas only 10 messages were posted 
on the New York Statewide Action Team listserv during the same period. 

By the summer of 1997, the Regional Alliance had consolidated its listservs (from 
70 to 29). A brief review of the postings on a sample of the listservs and a Consortium 
survey of subscribers suggests that usage has increased and the exchange of information 
is useful. Clearly, Regional Alliance staff have made an effort to regularly contribute to 
each of the listservs in order to help them become a valuable source of information for 
subscribers. 

Outcomes Reported by Participants 

The Consortia had done little to evaluate their technology services, partly because 
the Consortia’s use of technology was relatively new and still evolving. Several 




74 



88 



Consortia were planning to survey a sample of their users to assess impact, but, at the 
time of this evaluation, they had not yet detailed their plans for doing so. The most 
systematically collected data pertained to the number of teachers who had visited 
demonstration site computers and attended workshops. The Consortia collected and 
reported these numbers quarterly; for example, the Biodiversity demonstration site 
(WERC) reported that in one quarter 600 teachers attended six workshops that featured 
ENC materials, and 61 teachers stopped by the resource center to look for curriculum 
materials. 

The Consortia were beginning to collect data on the use of their Web sites. For 
example, the Midwest Consortium reported getting about 3,000 accesses (or “hits”) a 
week on the math and science pages on the Pathways to School Improvement site, and 
the Regional Alliance reported getting about two hits a minute on its home page. 
However, at that point, information on the use and impact of Consortia Web sites was 
still limited. 

By the summer of 1997, some of the Consortia had begun more systematic 
evaluations of their use of technology. As was mentioned in the preceding section, the 
Regional Alliance conducted a survey of some of its listserv subscribers. The 
overwhelming majority found the listservs “useful or relevant” and “shared list postings 
with colleagues.” 

The evaluators were able to gather some data from participant surveys. Among all 
participants surveyed, 29 percent had visited their Consortium’s home page and/or 
databases, and another 21 percent had visited the home page and/or databases maintained 
by the Eisenhower National Clearinghouse. Among those who visited Consortia home 
pages, between 80 percent and 90 percent rated the sites as “good” or “excellent.” In 
addition, the more intensive and repeated the participant’s contact with the Consortium, 
the more likely the participant was to have visited the home page. 

The survey also indicated that school staffs visited a Consortium Web site less than 
other groups of educators. Only 14 percent of teachers, 23 percent of school district 
officials, and 21 percent of intermediate service agency staff reported visiting a 
Consortium Web site. In contrast, 38 percent of state officials and 39 percent of those 
from higher education institutions reported doing so. It is reasonable to assume that 
differences in access to technology among these different groups explain some of the 
variation in the use of Consortia Web sites. 
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Strengths and Weaknesses of the Consortia’s Use of Technology 

The Consortia used technology, in varying degrees, to offer information about 
themselves, to disseminate curricular resources and information about school reform, to 
facilitate communications between groups of educators, and to promote the use of 
technology in classrooms. In looking across all 10 Consortia, the evaluation team found 
some promising examples of each of these uses. At the same time, it found some 
unresolved issues associated with each use of technology. 

Web Sites 

The Consortia’s Web sites demonstrated potential to advertise each Consortium’s 
activities and to direct users to curricular resources and information. As the use of the 
technology matures, the Consortia will need ways of collecting data that can help them 
better understand their efforts’ impact on clients. 

Some of the data can be collected automatically. It is possible to design a Web site 
that counts how many visits it received, how many visits there were to each of the site’s 
locations or pages, the date and time of each visit, the duration of time from one visit to 
the next, the IP (Internet Protocol) addresses of the machines that visited the site, and the 
address of the Web site visited just before a visit to the target Web site (i.e., the 
navigation route visitors took). This information would tell a Consortium how often its 
material was being used and which pages were used most often. A Consortium could 
also determine how frequently visitors entered its site from another Consortium, the 
ENC, or elsewhere. The ENC could determine what number and percentage of its visits 
were referred from one of the Consortia or the technology demonstration sites. Few 
Consortia kept such detailed records or analyzed them in this way. However, 
automatically collected data do not indicate which audience accessed the Web site, what 
use people made of the information, or how useful they found it. For that kind of 
information, the Consortia will need to use surveys or other methods. 

Teacher Access and Usability 

Another issue in assessing the Consortia’s use of technology is that 
telecommunication is a two-way street. No matter how much information the Consortia 
provide, the effectiveness of the effort will be limited if schools and classrooms do not 
have the telecommunication services with which to access the information. 
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In the fall of 1996, the National Center for Education Statistics (NCES, 1997) 
surveyed 911 public elementary and secondary schools about their access to 
telecommunication services. The study reported that 65 percent of U.S. public schools 
had access to the Internet. Although a large number of schools did not have access, their 
number was decreasingly rapidly; the percentage of schools that were connected 
increased from 35 percent to 65 percent in just 2 years. Of those schools that were not 
connected, 85 percent reported that they planned to be by the year 2000. 

It is important to note, however, that access to the Internet was not equally 
distributed across school systems. Only 53 percent of schools with large proportions of 
students from poor families had access. In addition, although the number of schools that 
had access was significant, the number of classrooms that were connected was still small. 
Only 14 percent of all instructional rooms (classrooms, labs, and library media centers) 
were on the Internet. This low level of classroom access means that teachers’ and 
students’ use of the Internet was still limited. 

However, there were signs that the situation was changing. As with schools, the 
number of classrooms that were connected to the Internet was growing quickly. The 14 
percent figure reported in 1996 was a fourfold increase since fall 1994, when only 3 
percent of all instructional rooms had access to the Internet. 

The type of connection that schools had also limited their use of the Internet. In fall 
1996, 74 percent of the schools that had access to the Internet reported using modems. 
This is the lowest form of bandwidth, or speed, at which materials can be downloaded. 
This is particularly problematic when using the World Wide Web. Thus, although a 
growing number of schools had some form of access to the Internet, relatively few had 
easy access to the World Wide Web. 

Linkages and Value Added 

By the spring of 1997, the Consortia had not yet fully linked their Web sites so that 
users who found a resource in one Consortium Web site were directed to similar 
resources in other Consortia or ENC sites. Such linkages represented a coordination 
challenge. As each Consortium continues to expand its Web site with curricular and 
other resources similar to those offered by other Consortia or the ENC, much closer 
coordination will be necessary to avoid redundancy and streamline user access. 

The issue of fuller linkages between Web sites raised the much broader question 
about the value added by the Consortia’s presence on the Web. The World Wide Web 



contains exciting, highly graphic information related to science and math. Are additional 
Web sites needed when there is so much high-quality information already available from 
the Exploratorium, the Smithsonian, NASA, dozens of NSF projects, and hundreds of 
universities? What are the unique contributions of the Consortia Web sites? 

Consortia directors argued that teachers did not always know about the availability 
of materials, how to locate them easily, or how to find them in a usable format. Consortia 
need to consider the value that their Web sites add to their efforts, keeping their focus on 
the unique or special purpose of their sites and on the most effective use of constrained 
resources 

Listservs and Other Communication Tools 

Earlier, this report described how some Consortia created listservs to promote 
networking and communication among educators. These listservs varied in their levels of 
activity. As Consortia directors who used listservs learned, listservs must be attended to 
if they are to attract attention and be active. That is, Consortia staff had to continually 
stimulate conversation among very busy educators. Just establishing a listserv did not 
guarantee communication and networking. 

As teachers’ access to the Web increases, the Consortia will be able to use their 
growing interactive capabilities. Technological developments already allow users to 
interact in real time over the Internet. These new interactive environments open up new 
and interesting ways for the Consortia to provide professional development services and 
technical assistance if they were to develop the tools and interactive materials for doing 
so. But, as with listservs, the newer interactive environments require additional staff time 
and attention. 

Supporting the Federal Role in Promoting the Use of Technology 

The federal role in promoting the use of technology in education is to provide “the 
momentum to support state and local efforts to meet the technology literacy challenge. 
This is done through leadership, targeted funding, and support for activities that will 
catalyze national action” (U.S. Department of Education, 1996). 

Recently, the Report to the President on the Use of Technology to Strengthen K-12 
Education in the United States (Panel on Educational Technology, 1997) made specific 
recommendations that further elaborate both the potential federal role and present a 
coherent strategy. To summarize, the report recommends that the federal strategy: 
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• Focus on learning with technology, not about technology. 

• Emphasize content and pedagogy, not just hardware. 

• Give special attention to professional development. 

• Engage in realistic budgeting. 

• Ensure equitable, universal access. 

• Initiate a major program of experimental research. 

The challenge for the Consortia as they continue to promote and use technology is 
to make sure that they are guided by these principles. Through their Web sites and 
professional development efforts, the Consortia can follow the Panel’s guidance and 
emphasize learning with technology, as well as content and pedagogy. As the Consortia 
provide technical assistance to states and districts, they can promote realistic budgeting 
and equitable, universal access. Finally, the Consortia can help disseminate research. 

In addition, the Consortia must coordinate their efforts with others involved with 
technology and education. For starters, the Consortia with the strongest technology 
capacity can lend expertise to the other Consortia. The Consortia have made significant 
progress in upgrading and linking their Web sites and promoting the ENC, and some 
have expanded their professional development activities on learning with technology. 
But better coordination and communication between Consortia staff involved in learning 
with technology could help all Consortia advance in this area. 

Many of the Consortia do collaborate and coordinate with other federal programs 
that support the use of technology. However, there are some inherent difficulties in such 
collaboration. As with Regional Technology in Education Consortia, some federal 
programs are short-lived, serve regions that don’t match the Consortia’s regions, and are 
housed in organizations that are potential competitors with the Consortia for federal 
funding. Despite these difficulties, more and stronger relationships across federal 
programs involved with technology in education would go a long way in supporting 
learning with technology. 
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VII. NETWORKING AND COORDINATION 



The statute authorizing the establishment of the Regional Consortia Program 
(P.L. 101-589, Title II) directed the Consortia to “if feasible, maintain on-line computer 
networks with all regional consortiums and the National Clearinghouse for Science and 
Mathematics Education Materials” (Sec. 2017). It contained no other reference to 
forming a national network. Nevertheless, as noted in the First Interim Report of this 
evaluation, “there was an expectation (in Congress) that the Consortia would be 
organized as a national network” (Haslam, Colopy, & Turnbull, 1996). Subsequently, 
under IASA, Congress made its expectation more explicit by saying that the Consortia 
may use their funds to “work cooperatively with the other regional consortia, the 
Eisenhower National Clearinghouse.. .and federally funded technical assistance providers 
to more effectively accomplish [their work]” (Sec. 13302). 

Early in the program’s development, the Consortia made progress in forming a 
national program identity through their networking activities (Haslam, Colopy, & 
Turnbull, 1996). Consortia directors and staff recognized, however, that despite potential 
payoffs of working together and with other assistance providers, their first order of 
business had to be building their own capacity and working out relationships with their 
host organizations. In addition, delays in organizing the Eisenhower National 
Clearinghouse (ENC) slowed an important dimension of their network-building activities. 
The Consortia have been actively involved in networking and coordination activities 
following the early years. In fact, the Consortia and the ENC proclaim their 
determination to work together as a network on an Internet home page, which identifies 
them as the “National Network of Eisenhower Regional Consortia and Clearinghouse.” 

This chapter reports the evaluators’ findings about the Consortia’s continuing 
networking and coordination efforts. Specifically, the chapter examines the Consortia’s 
collective efforts on: 

• The development of a common data collection system and program performance 
indicators. 

• The use of cross-Consortia committees and task forces. 

• Coordination with the Eisenhower National Clearinghouse. 

• Coordination with other assistance providers. 

• Coordination efforts with other federal education reform initiatives. 
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The chapter concludes with a summary of the strengths and weaknesses of the 
Consortia’s networking and coordination efforts. 

Development of the Consortia Data System and Program Performance 
Indicators 

The Consortia directors and the Consortia evaluators have concentrated a 
considerable amount of their attention on the development of the Consortia data system 
and program performance indicators. Discussions and work around these tasks have 
focused on identifying the core elements of what the Consortia do and how their work 
should be reported to OERI and to Congress. 

The Cross-Consortia Descriptive Data System (CCDDS) represents a significant 
investment by the Consortia in creating a common descriptive framework for reporting on 
their work in the aggregate to OERI and to Congress. As a review of the Consortia’s 
annual report suggests, individual Consortia also use CCDDS data to report on their own 
portfolio of activities. Several Consortia evaluators also reported that reviewing the 
CCDDS data helps Consortium staff reflect on their work and, from time to time, 
consider options for changing it. 

Early progress on the reporting system took longer than the Consortia expected — in 
part because of the difficulty of agreeing on the definition of an “activity” to guide the 
reports on participants, content, duration, and resources expended. As one Consortium 
director who was very active in these efforts in the spring of 1996 explained: 

Our mission for now is to describe what the 10 Consortia do and to help each 
Consortium to look at their distribution of activities.... We’ll see a big difference in 
approaches. I hope there will be a conversation as to why these differences exist. I 
hope this will be a good way to reflect on the program overall. 

As of January 1997, the CCDDS contained descriptive information on six categories of 
Consortia activities and services: product development, collaboration and network 
building, information dissemination, planning assistance, technical assistance, and 
training. 

The work on the program performance indicator system, carried out in conjunction 
with the Eisenhower Program Office in OERI, led the Consortia to extend the work on 
the CCDDS by focusing on program outcomes. The draft performance indicators, which 
were submitted for review by the Acting Deputy Secretary of Education in September 
1997, after more than 18 months of work by the Consortia and OERI, affirm the 





Consortia’s commitment to carrying out the broad set of responsibilities assigned to them 
by Congress and in the Department of Education’s priorities. The performance indicators 
are defined around six program objectives: 

• Provide high-quality technical assistance (including planning assistance, 
technical assistance, and training). 

• Provide continuing technical assistance to individuals who (a) will assist or train 
others and/or (b) will work with at-risk, underrepresented, and/or underserved 
populations. 

• Identify promising and exemplary practices in education. 

• In coordination with the ENC, disseminate information about promising and 
exemplary practices in mathematics and science education. 

• Contribute to collaboration and networking with targeted groups within and/or 
across states. 

• Contribute to the use of informal education entities (e.g., science and 
technology centers, museums, libraries, Saturday academies, 4H programs). 

The development of CCDDS and the program’s performance indicators have the 
potential of adding clarity and focus to the program’s identity and mission. In particular, 
the indicators set performance benchmarks that represent consensus among the Consortia 
about the particular outcomes they will strive to achieve, priorities for areas of work and 
types of individuals to target for Consortia activities, and strategies that will be used. 

Data collected recently also point to some positive benefits for individual 
Consortia’s evaluation efforts. For example, some Consortia annual reports draw heavily 
on the CCDDS and are beginning to address the performance indicators. At the same 
time, Consortia staff say that their internal evaluations of both quality and impact 
continue to rely on significant amounts of anecdotal evidence. 

Cross-Consortium Committees and Task Forces 

A second dimension of the Consortia’s collaborative activities has been the work of 
four committees or task forces. In the spring of 1997, these included: 

• The Communications Committee, which is responsible for preparing reports on 
the Regional Consortia Program and the ENC. 

• The Equity Task Force, which is compiling resources on equity issues as they 
pertain to mathematics and science education. According to the 1997 annual 
report of the Regional Alliance, the Equity Task Force completed work on the 
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Equity Resource Guide in January 1997 and expected to produce a CD-ROM by 
October 1997. 

• An Evaluation Committee, which works with the directors on the reporting and 
indicators systems and other issues related to program evaluation and will revise 
the survey instruments used in this evaluation and pilot test the performance 
indicators data collection and reporting systems in 1997-98. 

• The Professional Development Task Force, which is developing a CD-ROM 
toolkit on professional development. 

In all of these committees and projects, one Consortium assumes lead responsibility for 
the activity and for developing the related products. Other Consortia assign at least one 
member of their staff to the group or team. • 

Overall, the work of the committees and task forces appears to have taken 
advantage of economies of scale by producing one product useful to all 1 0 Consortia. In 
addition, individual Consortia have been able to take advantage of the large pool of 
expertise among all Consortia staff. 

Collaboration and Coordination with the Eisenhower National 
Clearinghouse 

The Consortia’s work with the ENC was another example of collaboration under 
way within the Regional Consortia Program. The Consortia and the ENC work together 
in a variety of ways. First, the Consortia directors meet regularly and communicate often 
about issues of concern to the Regional Consortia Program. The ENC director 
participates in most, if not all, of these meetings and other communications. Indeed, he 
served for a year as the chairperson of the group. The ENC is also active on the 
committees and task forces charged with developing products for dissemination and use 
by all of the Consortia and the ENC. In addition to participating on these committees, the 
Consortia have worked closely with the ENC to establish an on-line database of informal 
science education resources. According to 1997 Consortia reports, the database provides 
clients with descriptive information on over 1 ,200 organizations and centers across the 
nation offering informal science programs and activities. 

In addition, the Consortia work with the ENC to collect and disseminate 
information about mathematics and science education. This work became a fairly routine 
part of Consortia operations: the Consortia provided materials to the ENC and helped to 
publicize the ENC as a resource. The ENC, in turn, advised the Consortia on packaging 
materials. According to Consortium directors, all these activities required modest 
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investments of staff time and resources, although individuals with lead responsibility for 
organizing the groups and product development sometimes spent considerable time on 
these tasks. 

Coordination with Other Assistance Providers 

When the Consortia first began operations, they had a mandate to coordinate their 
work with state and local reform initiatives and other external providers. As reasonable 
as this mandate might have seemed, it was difficult to carry out. First, as noted earlier, 
the Consortia had to attend to their own internal development issues. Second, many 
major reform initiatives were already well under way, and, consequently, the Consortia 
were relatively late arrivals on the scene. Third, the mandate to coordinate was a one- 
way street. No federally sponsored technical assistance providers or other recipients of 
federal support for the reform of mathematics and science education had a mandate to 
work with the Consortia. 

This situation has changed. The Consortia are well on their way to becoming 
partners with other assistance providers. One of the most important changes is that the 
Regional Educational Laboratories, the new Comprehensive Technical Assistance 
Centers, and the Regional Technology in Education Consortia have mandates to 
coordinate their services with the Regional Consortia, among others. In the spring of 
1996, the new technical assistance centers were just beginning their work, and, like the 
Consortia 3 years ago, they were concentrating on building their organizations and 
marketing themselves in the field. Therefore, it was not surprising that the evaluators saw 
no evidence of extensive coordination of services between the Consortia and these other 
providers. There were, however, several promising starts. 

The host Regional Educational Laboratories of three Consortia — WestEd, which 
houses WERC; Appalachia Educational Laboratory (AEL), the host institution for 
Consortium at AEL; and the Southwest Educational Development Laboratory, where 
SCIMAST is located — were awarded grants to operate Comprehensive Technical 
Assistance Centers. The host educational laboratories for the Midwest Consortium and 
the SERVE Consortium — the North Central Regional Educational Laboratory (NCREL) 
and SouthEastem Regional Vision for Education (SERVE), respectively — received grants 
to form technology consortia. In two of these cases — the Consortium at AEL and the 
Midwest Consortium — there is some overlap in staff: the directors of both Consortia are 
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also assigned to the new technical assistance centers. As the director of the Consortium 
at AEL explained: 

This is a great arrangement for us and for our ability to work with the field. When 

somebody calls with a question or a request, we know exactly where to send 

them — usually just down the hall — and we have a lot more things to offer them. 

This is a lot less confusing for our clients and a lot more effective for us. 

The Midwest Consortium director added that locating these programs in the same 
institution made it easy to leverage resources from the technology center and from the 
laboratory’s technology program to support Consortium activities and services. 

These two examples illustrate the benefits of locating various technical assistance 
centers in the same host organization. They also suggest that these arrangements may be 
most effective when staff assignments overlap, particularly at the director level. What 
look like obvious opportunities for coordination to someone assigned to both 
organizations may appear less obvious and less easy when there is no overlap. 

In a third example of coordination, under very different circumstances, at least two 
Consortia were involved in nascent efforts to coordinate parts of their work with other 
technical assistance centers in their regions. The director of the High Plains Consortium 
(HPC) joined with the director and staff of the regional laboratory, the Mid-continent 
Regional Educational Laboratory, to host a meeting of representatives from the 1 1 other 
federally supported technical assistance centers that work in some portion of HPC’s 
service region. In a similar effort, the co-directors of the Regional Alliance began to meet 
with the directors of other technical assistance centers in the Northeast “to get to know 
each other, to look for ways that we can coordinate our services, and to discuss common 
problems and concerns.” This group met several times during spring 1996, and, 
according to one of the Regional Alliance’s co-directors interviewed at the time: “At this 
point, none of us is quite sure where this is going, but it is very clear that we need to 
continue talking and that there are some real advantages in working together.” According 
to the 1 997 annual report of the Regional Alliance, this group continues to meet. A 
listserv created and maintained by the Alliance facilitates its work. 

In the spring of 1 996, the co-directors of the Regional Alliance reported a fourth, 
more concrete example of a partnership being formed with another technical assistance 
provider. A Desegregation Assistance Center in the Northeast included support for the 
Regional Alliance’s Equity Steering Committee in a proposal to the Department of 
Education. The Equity Steering Committee, with some additional members chosen by 




86 93 



the Desegregation Assistance Center, were to act as the center’s advisory group. This 
role ensured the continuation of the Equity Steering Committee and represented an 
opportunity for coordination between the center and the Regional Alliance. 

Coordination with Other Federal Mathematics and Science Education 
Reform Initiatives 

All the Consortia have entered into a variety of other working relationships with 
other major federal mathematics and science education reform initiatives. Prominent 
among these partnerships are those with the National Science Foundation’s various 
systemic reform projects: the Statewide Systemic Initiatives (SSIs), the Urban Systemic 
Initiatives (USIs), and the Rural Systemic Initiatives (RSIs). The Consortia also became 
involved in supporting some of the Department of Education’s Eisenhower State 
Curriculum Frameworks Projects. Some Consortia focused their support for the 
development and dissemination of curriculum frameworks in states that did not receive 
Eisenhower State Curriculum Frameworks Project grants. The Consortia have assisted 
individual projects and facilitated several networks of framework projects. Consortia’s 
work with and support for other federal mathematics and science education reform 
initiatives has included, but certainly has not been limited to, the following: 

• Direct financial support for project activities or events (e.g., support for printing 
and disseminating a curriculum framework, support for convening a meeting). 

• Technical assistance in completing key project tasks or activities (e.g., 
assistance in planning or convening a meeting or other event). 

• Convening state, regional, and national meetings of projects to address common 
themes or tasks. 

Specific examples include the following: 

• In Maine, the Statewide Action Team organized and supported by the Regional 
Alliance earmarked $10,000 available to the team from the Alliance to support a 
pilot effort to implement a new state curriculum framework. Other state teams 
supported by the Regional Alliance have used Consortium resources for similar 
activities. 

• The Midwest Consortium has provided a variety of professional development 
and training to district staff involved in Urban Systemic Initiatives in Chicago, 
Columbus (Ohio), and Detroit. According to Consortium reports, topics of 
these activities included team building, the change process, and new curricula in 
mathematics and science education. In each site, the Consortium has provided a 
series of activities that have extended over periods ranging from several weeks 
to more than a year. 
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• The SSI Network sponsored by SCIMAST, and discussed in more detail in the 
earlier chapter on Consortium teams and networks, brought together directors 
and staff from SSI projects in the Consortium’s service region to discuss topics 
of common interest and concern related to the implementation and operation of 
their projects. 

According to Consortium reports, some of this assistance leveraged other kinds of 
assistance and resources. In other cases, Consortium resources were the glue that held 
otherwise disparate elements of a project together. Leaders and staff of these projects 
reported that Consortia networks and other forums presented important and useful 
opportunities for them to meet with colleagues from other projects and other states. In 
the Northeast, project directors have participated in all of the Regional Alliance’s teams 
and networks. In an interview conducted in the spring of 1997, one SSI project director 
explained the benefits of her participation for her state’s SSI project this way: 

We wanted to develop a handbook for disseminating our framework, but we really 
didn’t have the time or the money to develop it. Then, in a meeting sponsored by 
the Alliance, we found that another SSI project in our region had already developed 
a handbook like we needed. So, with only a few changes, we were able to borrow 
theirs for our state. This was really helpful to us. 

In this and other regions, SSI staff have typically served as members of their 
Consortium’s regional advisory boards and state teams. These affiliations give them 
opportunities to advise the Consortium on their needs and interests. Currently, most of 
the SSIs and State Curriculum Frameworks Projects are completed or nearly completed. 
Consortia partnerships with USIs and RSIs, which currently are a large part of the 
portfolio of the Midwest Consortium, are continuing. 

Data collected in the spring of 1996 and 1997 suggest that the Eisenhower State 
Programs and Goals 2000 have not appeared on the Consortia’s list of active 
collaborations. Eisenhower State Program staff have served on Consortia advisory 
boards and been members of state teams, but, as of spring 1997, only the Pacific 
Mathematics and Science Consortium (Pacific Consortium) reported any extensive 
involvement with an Eisenhower State Program. In this case, the Consortium links its 
programs with programs sponsored by the Eisenhower State Program to “ensure that 
projects are building on each other and to make sure Visions and Dreams [a Consortium 
program] steps in for worthwhile projects that Eisenhower can’t fund.” Other Consortia 
reported more limited examples of work with Eisenhower State Programs. The diffuse 
nature of the Goals 2000 projects in the states may account for the lack of coordination 
opportunities available to the Consortia. 
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Strengths and Weaknesses of Consortia Networking and Coordination 

Consortia efforts to work together and with others have had some important 
payoffs. In each of the examples described in this chapter, the evaluators found clear 
benefits of the Consortia’s networking and coordination efforts. 

The development of the Regional Consortia Program’s performance indicators and 
the CCDDS has contributed — and may continue to contribute — to the Consortia’s 
capacity to reflect on, describe, and assess the impact of their activities and services. In 
addition to benefiting the Consortia, these systems are also likely to help ED monitor the 
work of individual Consortia, as well as the Regional Consortia Program itself. 

Cross-Consortia task forces and committees developed products to help all the 
Consortia achieve economies of scale and allow them to capitalize on the strengths of 
other Consortia. As one director put it, “For a relatively small investment, we get a lot of 
valuable tools and products that we can use in our regions.” 

The creation of the professional development CD-ROM toolkit, which is led by 
staff at the Midwest Consortium, illustrates the potential of these collective ventures to 
enhance the quality of Consortia activities and services while achieving some economies 
of scale. As the discussion on Consortia uses of technology in the preceding chapter 
described, the Midwest Consortium for Mathematics and Science Education (Midwest 
Consortium) and its host organization, the North Central Regional Educational 
Laboratory, have considerable production capacity and experience from the development 
of Pathways to School Improvement. By taking advantage of this capacity, other 
Consortia can obtain a higher-quality toolkit than they might have created on their own 
and without heavy investments in developing their own capacity. 

Collaboration has also served as a professional development vehicle for Consortia 
staff. Some networking activities, such as the Equity Task Force and the Professional 
Development Task Force, gave at least some Consortia staff with similar interests the 
chance to work together. 

In the spring of 1996, the Consortia’s relationships with new federally sponsored 
technical assistance providers were just beginning to develop. These will take time to 
mature. Several early examples point to the advantages of housing these providers in the 
same organization and of some overlap in staff. One advantage is the relatively easy 
coordination of service delivery; a second advantage is the ability to leverage resources 
from one provider to serve the clients of another. These advantages can be readily 
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achieved when staff have assignments in both organizations and when they have authority 
to allocate the resources of both organizations. These advantages may be much more 
difficult to achieve when there is no overlap in staff or when the providers have different 
organizational homes. 

Consortia partnerships with other organizations engaged in mathematics and 
science education reform represented opportunities for strategic investments of Consortia 
resources. Consortia partnerships with NSF’s various systemic reform initiatives and 
some of the Eisenhower State Curriculum Frameworks Projects have helped these 
projects complete their work and have made it possible to do things that would not have 
been possible under their original grants. The Consortia have not been as successful in 
forging working relationships with Eisenhower State Programs (except in the Pacific 
region) or Goals 2000 programs. 

As in other areas of their work, the Consortia need to be strategic in their selection 
of partnerships and networking opportunities. Given their limited resources to address 
complex issues, the challenge to the Consortia has been to weigh these opportunities 
against each other before investing and to assess the impact once the investments have 
been made. 
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VIII. CONCLUSIONS 



The purposes of this national evaluation of the Eisenhower Regional Consortia 
Program have been to describe the organization and evolution of the program and the 
Consortia themselves and to analyze the contributions that the Consortia have made to 
the improvement of mathematics and science education. This report has focused on the 
quality and impacfof activities and services that were prominent in the Consortia’s 
portfolios of assistance to their regions and the nation. These activities and services have 
included professional development, support for teams and networks, dissemination of a 
volume on promising practices in mathematics and science education, and, more recently, 
intensive work with selected local sites. The latter set of activities, which frequently 
included a variety of professional development, training, and technical assistance, 
represents the Consortia’s response to an invitational priority set by the Department for 
the second grant period. In addition, the evaluators reviewed the Consortia’s varied uses 
of technology, as well as the collaborative work among the Consortia, the Eisenhower 
National Clearinghouse, and other active participants in the mathematics and science 
education reform arena. The sample of activities included in this evaluation represent 
only a portion of the activities and services provided by the Consortia. The full set is 
actually considerably larger. However, based on reviews of Consortia reports and 
interviews with Consortia staff, the evaluators concluded that these activities are 
generally representative of the full range of activities and services provided by the 
Consortia during the first 5 years of operation. This concluding chapter reviews the key 
findings from the national evaluation and presents the evaluators’ general assessment of 
the Consortia’s contributions to mathematics and science education, as well as 
evaluators’ observations about the continuing evolution and maturation of the program. 

Evolution and Maturation of the Consortia 

From the outset, the Consortia have faced the challenges of carrying out an 
ambitious agenda of service to their regions and building and maintaining the capacity to 
provide those services. As reported in the First Interim Report on this evaluation, the 
Consortia devoted considerable time and energy in the first few years of their existence to 
building their individual and collective capacity and to establishing solid working 
relationships among themselves and with key stakeholders, constituents, and potential 
participants and recipients of their activities and services (Haslam, Colopy, & Turnbull, 
1996). Near the end of the program’s third year, the evaluators concluded that the 
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Consortia had “generally succeeded in this critical task.” Given that these were new 
organizations and that they faced a crowded and not always welcoming reform area, this 
was no small accomplishment. 

By the spring of 1997, the Consortia’s portfolios of activities and services had 
become relatively stable, suggesting that the Consortia have matured as organizations, 
and along with the ENC, as a national program. However, the Consortia’s main 
functions and the list of authorized activities are very extensive. In choosing among 
various options for carrying out these functions and engaging in at least some of the 
authorized activities, the Consortia have been influenced by demands — which are not 
always clear or even very stable — from the state and local levels and by their own 
institutional capacities to do some things and not others. In addition, Consortia choices — 
like the choices made by many federal grantees — are influenced by the Department’s 
invitational priorities and requests for services, which provide strong messages about the 
Department’s expectations. As a result of these myriad factors that influence their 
choices, the Consortia have provided a wide range of activities and services to very 
diverse groups. 

The Department’s invitational priority to concentrate at least some resources and 
services in work with intensive sites, which was included in the solicitation for the 
second funding period, posed a real challenge to the Consortia as they tried to think 
strategically about how to maximize their impact. Given their relatively modest 
resources, should they invest in activities that reach large numbers of people in limited 
ways or concentrate on serving fewer people and programs more intensively? As the 
Consortia that are providing these intensive services are learning, these services are 
expensive and make significant demands on Consortia resources, including both funds 
and staff. At the same time, the Consortia appear to have concluded that they must 
continually try to maintain a balance between the more intensive services and less 
intensive services as they look for opportunities in their states and regions. 

The evaluators conclude that, in the future, it will be important for the Department 
to continue to work with the Consortia to maintain this balance and to preserve their 
options to pursue promising opportunities. Among other things, this will entail 
recognizing the balance between the Department’s priorities and expectations and those 
of the states and of the Consortia’s regional boards and providing the Consortia with the 
latitude to make choices. 
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The Contributions of Consortia Activities 

Across all the events, products, and working relationships for which the evaluation 
team collected data, participants and others who were familiar with the work of the 
Consortia generally praised the Consortia. In the spring of 1 996, substantial numbers of 
survey respondents credited their participation in Consortia activities with making at least 
a “moderate” contribution to their knowledge, skills, and professional activities. In many 
instances, participants called the contribution “major” with regard to changes in their 
knowledge, skills, and individual or organizational behavior. These perspectives were 
also echoed in interviews conducted throughout the evaluation. 

As of the spring of 1996, the evaluators concluded that the following activities and 
strategies stood out as particularly effective in achieving their purposes: 

• Professional development activities, when the Consortia worked with 
individuals and organizations that were themselves poised to deliver services to 
others or to engage in organizational change. 

• A package of grants, training, and follow-up with organizations that themselves 
start from a base of commitment and capacity to support instructional 
improvement. 

• Financial support and substantive contributions to teams and networks that have 
a clear sense of their purpose. 

• Creation of a new and welcome forum for communication among individuals 
active in reform who are the members of almost every team and network. 

In addition, this list should include the Consortia’s partnerships and collaborations 
among themselves and with others. For the Consortia and the Eisenhower National 
Clearinghouse, the partnerships have resulted in increased capacity to provide services at 
the national, regional, and state levels. For other partners, these relationships have 
yielded valuable resources and assistance in carrying out their work. An important 
characteristic of most, although perhaps not all, of these activities is the extent to which 
the Consortia provided more than one type of resource or used their support to leverage 
additional resources to support the participants, or both. The evaluators also found an 
area with considerable potential: the Consortia’s use of technology, which by 1997 was 
increasingly sophisticated. 

Also, in the spring of 1996, the evaluators concluded that on the basis of the survey 
and other data collected at the time, it was too early to determine the effectiveness and 
impact on other interesting and potentially valuable activities. One of these was long- 
term direct work with teachers in study groups, which had stimulated professional 



conversations and experiments with new classroom practices, but had so far transformed 
instruction in few, if any, classrooms. 

Finally, some activities appeared to have more limited impact and suggested the 
need for a review of their purpose and organization. For example, the dissemination of 
the book of promising practices was successful as a straightforward dissemination 
activity, but recipients who remembered receiving the book reported very little change in 
behavior. Data from the evaluation suggest that if the larger goal of the activity is to 
effect change in individual or organizational practice, some investment, albeit a modest 
one, in follow-up will increase the impact. Consortia support for state teams in states 
with ambitious and complex reform initiatives or in states in which state priorities and 
leadership are not well defined is a second example. In these cases, the support may have 
limited payoff until there have been extensive discussions about the roles that Consortia 
support can play and how these roles will address the new and evolving priorities. 

Looking across these findings and more recent data collected in the spring of 1 997, 
the evaluation team concludes that the Consortia did indeed add something of value to the 
nation’s mathematics and science education reform efforts. In some cases, the value was 
added in terms of reported increases in knowledge and skills. In other cases, the value 
was added by increased opportunities to communicate and collaborate with colleagues 
and peers who shared common concerns and tasks. At the institutional level — including 
state and local reform initiatives — value was added when Consortia resources and other 
assistance facilitated the next step in a task or project — possibly one that was not in the 
original plan or budget but that became important later. 

Experiences in the more successful activities point to the programmatic advantages 
of working with people and programs who are committed to and poised to make changes 
or improvement. Careful targeting and reaching agreements on long-term efforts and the 
availability of other resources have obvious payoffs. Initial experiences in at least some 
of the intensive sites also point to the importance of gaining commitments from the sites 
to engage in improvement efforts and to work with the Consortia in doing so. Both of 
these lessons are reflected in the “Strategies” included in the program’s new performance 
indicator system. 

These important lessons about targeting beg the question about working in the 
neediest sites. Experience in sites that are willing to commit to trying to make 
improvements and to working with the Consortia but that have made little or no progress 
on their own and that have very limited capacity to do so, points to the need for 
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substantial investments of time and money just to get started. The nine low-performing 
schools assisted by the Consortium at AEL are a good example. In these sites — sites that 
are probably most in need of help — there is a risk of failure or at least very slow initial 
progress. The Consortium at AEL and other Consortia have decided that the potential 
benefits of this work make the risk of failure or limited progress acceptable. It will be 
important for the Consortia and the Department to encourage and support this work by 
setting and maintaining their expectations accordingly. 

Challenges for the Future 

By the spring of 1 997, the Consortia had matured as organizations, both in terms of 
their capacity to provide a variety of services and to work in concert with a variety of 
other players in mathematics and science reform. Accordingly, since the early years of 
the program, the individual Consortium portfolios have taken on a clearer focus. These 
are important accomplishments, particularly in the face of the challenges the Consortia 
faced from the beginning of the program. The continued development of the Consortia 
reporting system and of the Regional Consortia Program’s performance indicator system 
have the potential of adding more clarity and focus to the program’s purpose, particularly 
as future national evaluations and the Consortia’s own internal evaluations report on the 
Consortia’s performance against the new performance indicators. 

Clarity of purpose and focus is critical because the Regional Consortia Program and 
the individual Consortia continue to confront the difficult choices inherent in the omnibus 
character of the program’s charge and priorities from OERI and Congress, as well as the 
diverse needs and priorities of their regions. The Consortia have tried to do many things, 
but they are not big enough — in terms of staff and resources — individually or 
collectively, to do everything in improving mathematics and science education. At the 
same time, the program is neither small enough nor yet secure enough to pick a tightly 
specified mission and stick to that mission. To the extent that this tension informs an 
ongoing dialogue about what the Consortia can and should do and guides thoughtful 
decisions about priorities, it will have a beneficial effect on the program. 

Thus, the overall value added and obvious maturation of the Consortia 
notwithstanding, the final conclusion of this evaluation is that important challenges lie 
ahead — challenges to be addressed by the Consortia, by OERI, and by Congress as they 
look ahead to the future of the program. When viewed in the context of the ambitious 
goals for the Regional Consortia Program, the current annual funding level of $15 million 
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looks like a relatively modest investment in solutions to large and complex issues. 
Consortia collaborations to achieve economies of scale and to take full advantage of the 
special capacities of individual Consortia coupled with assistance strategies that leverage 
resources from other institutions and programs have helped stretch these limited 
resources. These observations as well as similar observations about elements of the 
Consortia’s work point to a recurring theme in the evaluation findings: the importance of 
making strategic choices about investing limited resources to achieve maximum impact. 
Therefore, it is incumbent on both OERI and the Consortia to continue seeking clarity 
about the program’s mission and how well that mission is carried out, to set appropriate 
expectations for the quality, scope, and impact of the Consortia’s portfolios of activities 
and services, and to monitor the extent to which those expectations are met. 
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SURVEY PROCEDURES AND INSTRUMENTS 
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SURVEY PROCEDURES 



Survey Development 

The evaluation team developed three questionnaires to collect data on 
participant/recipient satisfaction with three categories of Consortia activities and services: 
professional development, support for teams and networks, and dissemination of 
information about promising practices. The instruments are included as Exhibits A-l, 
A-2, and A-3 at the end of this appendix. 

The instruments include a set of common items about respondents’ occupation, 
participation in the activity, and ratings of the quality, utility, and uniqueness of the 
activity or service. The questionnaires also include items about participants’ visits to 
home pages and electronic databases created by the Consortia. Questionnaires were 
tailored for each activity category and for each activity within the category. 

Draft questionnaires were reviewed by Consortia staff and ED staff. As 
appropriate, reviewers’ comments and suggestions were incorporated into the final 
versions of the instruments. 

Sampling Procedures 

In March and April 1996, the evaluators contacted Consortium directors and other 
staff to identify professional development activities and active teams and networks and to 
determine whether individual Consortia had distributed the publication Promising 
Practices in Mathematics and Science during the past 12 months and whether they could 
provide a list of recipients. The general criteria for including a professional development 
activity in the survey sample were that it (1) represented a relatively large investment of 
Consortium resources, (2) was intensive, (3) included some kind of follow-up, and (4) 
was conducted during the current grant period. The sample of state teams includes at 
least one state team from each region in which a Consortium supports state teams, and for 
regions with large numbers of states or in which support for state teams is a signature 
activity, it includes more than one team. The survey sample of networks includes all but 
one of the active networks supported by the Consortia. The sample of recipients of the 
Promising Practices publication came from mailing lists that were readily available from 
five of the Consortia. 
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This process yielded 25 lists of participants and recipients of Consortia activities 
and services in three categories. Names without addresses and names of inappropriate 
respondents (e.g., ED staff) were deleted from the lists. 

For participant and recipient lists with 70 or fewer names, surveys were sent to the 
entire population. For participant and recipient lists with more than 70 names, the 
evaluators drew a simple random sample to accommodate the differences in the numbers 
of participants/recipients. The one exception to this procedure is the publication recipient 
list provided by the Midwest Consortium, which included 1,226 names. Twenty percent 
of this population was sampled to obtain a sample size of 245. The evaluators chose this 
sample size because it brought the size of the total sample in this category to a number 
close to the original projection of 500 respondents in this category. 

The evaluators weighted the survey data on the basis of the relationship between the 
sample size and total number of participants or recipients in the activity. Thus, for survey 
responses from participants in activities from which samples were drawn, the evaluators 
added data weights that are inversely proportional to the sample size and the total number 
of participants in the activity. For example, the survey sample included 28 of the 70 
participants on the Pennsylvania State Team membership list supplied by the Mid- 
Atlantic Consortium. Taking the inverse proportion of the size of the sample and the 
population, a weight of 2.5 was assigned to responses from members of this team. The 
sample weights are reflected in the estimates of aggregate responses within each of the 
three activity categories. They are not reflected in the presentation of data on individual 
activities in each category. 




Survey Administration and Follow-up 

The evaluation team administered the surveys in two waves during May 1996. This 
schedule was determined by the availability of lists of potential respondents provided by 
the Consortia. The evaluators called all nonrespondents 2 weeks after the questionnaires 
were mailed to encourage them to complete and return the questionnaires as soon as 
possible. Most of these calls resulted in messages being left on answering machines or 
with people other than the respondents who took the call. In some cases, callers took 
responses over the phone. Copies of the questionnaires were mailed or faxed to all 
respondents who said that they had lost or not received the original. 

After the first round of follow-up calls, the response rates for each activity and for 
the three activity categories were recorded. For activities with response rates at or above 
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70 percent after the first follow-up, there was no additional attempt to contact 
nonrespondents. For activities with response rates below 70 percent, there was a second 
round of follow-up calls. For each of these activities, follow-up efforts ended when the 
response rate reached 65 percent or when callers had attempted to reach each 
nonrespondent at least twice. Overall, the follow-up efforts yielded response rates of 65 
percent or higher for 14 of the activities and overall response rates of 76 percent for 
professional development activities, 68 percent for teams, 72 percent for networks, and 
56 percent for dissemination of the Promising Practices publication. 



Exhibit A-1 



CUSTOMER SURVEY 

EISENHOWER MATH AND SCIENCE REGIONAL CONSORTIA 
PROFESSIONAL DEVELOPMENT ACTIVITY 
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CUSTOMER SURVEY 

EISENHOWER MATH AND SCIENCE REGIONAL CONSORTIUMS 
PROFESSIONAL DEVELOPMENT ACTIVITY 



Circle the number to the right to indicate your response for each question below. 

1. Please indicate the type of agency for which you work: (CIRCLE ONE) 

a. School 1 

b. School district 2 

c. Intermediate education agency or BOCES 3 

d. State education agency 4 

e. Institution of higher education 5 

f. Other (specify): . 6 



2. Which of the following activities are major parts of your professional responsibilities? 
(CIRCLE ALL THAT APPLY) 

a. Teaching elementary or secondary school students 1 

b. Teaching undergraduate or graduate school students 1 

c. Administering local education program(s) or organization(s) 1 

d. Administering State education program(s) or organization(s) 1 

e. Providing professional development or technical assistance 1 

f. Other (specify): . l 



3. What percent of your time is explicitly directed toward mathematics, science, or technology 
education? (CIRCLE ONE) 

a. Less than 25 percent 1 

b. 25 to 50 percent 2 

c. More than 50 percent, but less than 100 percent 3 

d. 100 percent 4 

Did you attend the [Professional Development Activity] sponsored by the [Consortium]? 

a. Yes 1 

b. No (END SURVEY AND RETURN TO PSA) 2 

c. Unsure (END SURVEY AND RETURN TO PSA) 3 



OMB No. 1880-0529. Approval expires 5/31/96. Reporting on this survey is authorized by law (20 
USC 8671). Your responses will be kept strictly confidential. Results from this survey will appear in 
summary or statistical form only, so that individuals cannot be identified. 

M6 



5. What was the extent of your involvement with the activity? (CIRCLE ONE) 

a. Attended everything 1 

b. Attended more than half of total 2 

c. Attended less than half of total 3 

d. Attended very little or nothing 4 

6. What were your purposes in participating in this activity? (CIRCLE ALL THAT APPLY) 

a. Generally keeping current professionally 1 

b. Obtaining ideas or contacts that would strengthen my work 1 

c. Obtaining ideas or contacts that would strengthen my 

organization’s existing work 1 

d. Obtaining ideas or contacts that would help my organization carry out a major, 

planned improvement effort 1 

e. Being a resource to others . . . 1 

f. Other (specify): • 1 

7. How fully did the activity serve your purposes? (CIRCLE ONE) 

a. It served my purposes completely 1 

b. It was a good start 2 

c. It was a start, but it did not provide sufficient information or 

guidance to enable me to follow up 3 

d. It did not serve my purposes 4 

8. Based on your experience, please indicate whether you think the activity is a unique resource. 

a. Yes 1 

b. No 2 

If you answered "Yes" please elaborate briefly 



Please indicate the type and extent of the follow up, if any, that [Consortium] staff have initiated with 
you. (CIRCLE ONE ON EACH LINE) 





Never 


Once 


More than 
once 


a. Followed up by phone 


1 


2 


3 


b. Followed up with workshops, training, meetings 


1 


2 


3 


c. Sent additional materials 


1 


2 


3 


d. Visited my workplace 


1 


2 


3 


e. Provided financial support 


1 


2 


3 


f. Other (specify): 


1 


2 


3 


Please indicate the type and extent of the follow up, if any, that has been available from 


an 


organization(s) other than the [Consortium] (CIRCLE ONE ON EACH LINE) 






Never 


Once 


More than 
once 


a. Followed up by phone 


1 


2 


3 


b. Followed up with workshops, training, meetings 


1 


2 


3 


c. Sent additional materials 


1 


2 


3 


d. Visited my workplace 


1 


2 


3 


e. Provided financial support 


1 


2 


3 


f. Other (specify): 


1 


2 


3 



Have you visited the home page or other electronic databases created by this consortium, 
other Eisenhower Regional Consortiums, or the Eisenhower National Clearinghouse? 
(CIRCLE ALL THAT APPLY) 



a. Yes, I have visited the home page and/or databases created by this 

consortium j 

b. Yes, I have visited the home page and/or databases created by other 

regional consortiums j 

c. Yes, I have visited the home page and/or databases maintained by the 

Eisenhower National Clearinghouse 1 

d. No, I have not visited any of these electronic sites, but I am aware that they 

exist (SKIP TO 14) 1 

e. No, I have not visited any of these sites; I did not know they 

existed (SKIP TO 14) 1 



O 
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12. Did attending [Professional Development Activity] lead you to visit one or more of these sites? 



a. Yes 1 

b. No 2 



13. Compared to other electronic sites you have visited, how would you rate the quality of 
these sites in each of the following areas? (CIRCLE ONE ON EACH LINE) 







Poor 


Fair 


Good 


Excellent 


a. 


Access through the Internet 


1 


2 


3 


4 


b. 


Access from one site to another 


1 


2 


3 


4 


c. 


Format of information included at the site 


1 


2 


3 


4 


d. 


Usefulness of the information at the site 


1 


2 


3 


4 



14. What effect has the activity had on your own work related to this topic? (CIRCLE ALL THAT APPLY) 



a. It confirmed what I was already doing 1 

b. I have considered doing something new in my job 1 

c. I have started to try something I learned from this activity 1 

d. I have incorporated something I learned from this activity into my job 1 

e. I have shared the ideas with a colleague(s) 1 

f. It has not had an effect on how I do my job 1 

g. Other (specify): • 1 



15. How large a contribution to your skills and knowledge has this activity made within each of the following 
domains? (CIRCLE ONE ON EACH LINE) 





None 


Small 


Moderate 


Major 


Mathematics/science/technology education 


1 


2 


3 


4 


Professional development 


1 


2 


3 


4 


Assessment/standard setting 


1 


2 


3 


4 


Planning 


1 


2 


3 


4 


Collaboration/communication 


1 


2 


3 


4 


Other (specify) : 


1 


2 


3 


4 
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16. 



How closely is this activity related to broader (e.g., local, state, or regional) mathematics, science, and 
technology reform efforts with which you are familiar? (CIRCLE ONE) 



a. It is closely related to broader reform efforts 1 

b. It is somewhat related to broader reform efforts 2 

c. It is not related to broader reform efforts 3 

d. I don’t know; I am not involved in broader reform efforts 4 



What effect, if any, has the [Professional Development Activity] had on the way your organization does 
its work? (CIRCLE ONE ON EACH LINE) 



None Small Moderate Major 



a. Helped generate awareness of new information 

within the organization 1 

b. Supported an ongoing program, policy, or practice 

of the organization 1 

c. Helped the organization initiate or establish a 

new program, policy, or practice 1 

d. Has not had an effect on the organization 1 



2 3 4 

2 3 4 

2 3 4 

2 3 4 



18. Compared with other similar activities, how would you rate the quality of this one in each of the following 
areas? (CIRCLE ONE ON EACH LINE) 



a. Accuracy of content 

b. Completeness 

c. Presentation 

d. Depth 



Poor Fair 



Good Excellent 



1 

1 

1 

1 



2 3 4 
2 3 4 
2 3 4 
2 3 4 



THANK YOU FOR YOUR ASSISTANCE 



PLEASE RETURN THIS SURVEY AS QUICKLY AS POSSIBLE, PREFERABLY WITHIN ONE WEEK, 

IN THE ATTACHED ENVELOPE. 

Policy Studies Associates 
1718 Connecticut Avenue, NW Suite 400 
Washington, DC 20009 
Attention: Sara Nathanson 

(202) 939-9780 
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Exhibit A-2 



CUSTOMER SURVEY 

EISENHOWER MATH AND SCIENCE REGIONAL CONSORTIA 

TEAM OR NETWORK 



CUSTOMER SURVEY 

EISENHOWER MATH AND SCIENCE REGIONAL CONSORTIUMS 

TEAM OR NETWORK 



Circle the number to the right to indicate your response for each question below. 

1 . Please indicate the type of agency for which you work: (CIRCLE ONE) 

a. School 1 

b. School district 2 

c. Intermediate education agency or BOCES 3 

d. State education agency 4 

e. Institution of higher education 5 

f. Other (specify): . 6 



2. Which of the following activities are major parts of your professional responsibilities? 
(CIRCLE ALL THAT APPLY) 

a. Teaching elementary or secondary school students 1 

b. Teaching undergraduate or graduate school students 1 

c. Administering local education program(s) or organization(s) 1 

d. Administering State education program(s) or organization(s) 1 

e. Providing professional development or technical assistance 1 

f. Other (specify): . 1 



3. What percent of your time is explicitly directed toward mathematics, science, or technology 
education? (CIRCLE ONE) 

a. Less than 25 percent 1 

b. 25 to 50 percent 2 

c. More than 50 percent, but less than 100 percent 3 

d. 100 percent 4 

4. Are you a member of the [Team or Network] sponsored by the [Consortium] 

a. Yes 1 

b. No (END SURVEY AND RETURN TO PSA) 2 

c. Unsure (END SURVEY AND RETURN TO PSA) 3 



OMB No. 1880-0529. Approval expires 5/31/96. Reporting on this survey is authorized by law (20 
USC 8671). Your responses will be kept strictly confidential. Results from this survey will appear in 
summary or statistical form only, so that individuals cannot be identified. 
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What has been the extent of your involvement in the [Team or Network] activities? (CIRCLE ONE) 



a. Essentially uninvolved 1 

b. Available if they need me 2 

c. Interested and involved in the activities 3 

d. Critical to the activities’ success 4 

What were your purposes in participating in this activity? (CIRCLE ALL THAT APPLY) 

a. Generally keeping current professionally 1 

b. Obtaining ideas or contacts that would strengthen my work 1 

c. Obtaining ideas or contacts that would strengthen my 

organization’s existing work 1 

d. Obtaining ideas or contacts that would help my organization carry out a major, 

planned improvement effort 1 

e. Being a resource to others 1 

f. Other (specify): . 1 



How fully did the activity serve your purposes? (CIRCLE ONE) 

a. It served my purposes completely 1 

b. It was a good start 2 

c. It was a start, but it did not provide sufficient information or 

guidance to enable me to follow up 3 

d. It did not serve my purposes 4 

Based on your experience, please indicate whether you think the activity is a unique resource. 

a. Yes 1 

b. No 2 

If you answered "Yes" please elaborate briefly 



Please indicate the type and extent of the follow up, if any, that the [Consortium] staff have initiated 
with you. (CIRCLE ONE ON EACH LINE) 











More than 






Never 


Once 


once 


a. 


Followed up by phone 


1 


2 


3 


b. 


Followed up with workshops, training, meetings 


1 


2 


3 


c. 


Sent additional materials 


1 


2 


3 


d. 


Visited my workplace 


1 


2 


3 


e. 


Provided financial support 


1 


2 


3 


f. 


Other (specify): 


1 


2 


3 



What effect has the activity had on your own work related to this topic? (CIRCLE ALL THAT 
APPLY) 



a. It confirmed what I was already doing 1 

b. I have considered doing something new in my job 1 

c. I have started to try something I learned from this activity 1 

d. I have incorporated something I learned from this activity into my job 1 

e. I have shared the ideas with a colleague(s) 1 

f. It has not had an effect on how I do my job 1 

g. Other (specify): . l 



How large a contribution to your skills and knowledge has this activity made within each of the 
following domains? (CIRCLE ONE ON EACH LINE) 

None Small Moderate Major 



a. 


Mathematics/science/technology education 


1 


2 


3 


4 


b. 


Professional development 


1 


2 


3 


4 


c. 


Assessment/standard setting 


1 


2 


3 


4 


d. 


Planning 


1 


2 


3 


4 


e. 


Collaboration/communication 


1 


2 


3 


4 


f. 


Other (specify): 


1 


2 


3 


4 



How closely is this activity related to broader (e.g., local, state, or regional) mathematics, science, 
and technology reform efforts with which you are familiar? (CIRCLE ONE) 



a. It is closely related to broader reform efforts 1 

b. It is somewhat related to broader reform efforts 2 

c. It is not related to broader reform efforts 3 

d. I don’t know; I am not involved in broader reform efforts 4 



13. 



What effect; if any, has the activity had on the way your organization does its work? (CIRCLE ONE 
ON EACH LINE) 

None Small Moderate Major 



a. Helped generate awareness of new information 

within the organization 1 

b. Supported an ongoing program, policy, or practice 

of the organization 1 

c. Helped the organization initiate or establish a 

new program, policy, or practice 1 

d. Has not had an effect on the organization 1 



2 3 

2 3 



2 3 

2 3 



4 

4 

4 

4 



14. Have you visited the home page or other electronic databases created by this consortium, 
other Eisenhower Regional Consortiums, or the Eisenhower National Clearinghouse? 
(CIRCLE ALL THAT APPLY) 



a. Yes, I have visited the home page and/or databases created by this 

consortium 1 

b. Yes, I have visited the home page and/or databases created by other 

regional consortiums 1 

c. Yes, I have visited the home page and/or databases maintained by the 

Eisenhower National Clearinghouse 1 

d. No, I have not visited any of these electronic sites, but I am aware that they 

exist (SKIP TO 17) 1 

e. No, I have not visited any of these sites; I did not know they 

existed (SKIP TO 17) 1 



15. Has your participation on the [Team or Network] led you to visit one or more of these sites? 



a. Yes 1 

b. No 2 



16. Compared to other electronic sites you have visited, how would you rate the quality of 
these sites in each of the following areas? (CIRCLE ONE ON EACH LINE) 







Poor 


Fair 


Good 


Excellent 


a. 


Access through the Internet 


1 


2 


3 


4 


b. 


Access from one site to another 


1 


2 


3 


4 


c. 


Format of information included at the site 


1 


2 


3 


4 


d. 


Usefulness of the information at the site 


1 


2 


3 


4 
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17. Where does the leadership for the current [Team or Network] activities come from? (CIRCLE 
ONE) 



a. Primarily from the Consortium 1 

b. Primarily from the members 2 

c. Equally balanced between Consortium and members 3 

d. I don’t know 4 



18. Listed below are some possible contributions that a Consortium can make to a network or team. 
Please indicate the effect of the contributions made by [Consortium] to the work of the [Team or 
Network], 

(CIRCLE ONE ON EACH LINE) 









Does 


Does 


Does 


Does 








with 


with 


with 


with 






Does not 


no 


small 


moderate 


major 






do 


effect 


effect 


effect 


effect 


a. 


Provided financial or other support for participants 


1 


2 


3 


4 


5 


b. 


Convened and facilitated meetings 


1 


2 


3 


4 


5 


c. 


Paid for the activities of the [Team or Network] 


1 


2 


3 


4 


5 


d. 


Contributed staff time to organizing the 














[Team or Network] 


1 


2 


3 


4 


5 


e. 


Produced and disseminated a [Team or Network] 














product 


1 


2 


3 


4 


5 


f. 


Provided substantive advice and guidance 


1 


2 


3 


4 


5 



19. Please indicate which of the following represent significant outcomes of the work of the 
[Team or Network]; (CIRCLE ALL THAT APPLY) 

a. Opened important new lines of communications 

b. Provided a unique forum for discussing important issues 

c. Created productive new working relationships 

d. Developed a needed product (e.g., standards for mathematics and science 

education, policy options, a plan) 

e. Supported worthwhile projects related to reforms in mathematics and science 

education 

f. There were no important outcomes 

g. Other (specify): 



1 

1 

1 

1 

1 

1 

1 
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THANK YOU FOR YOUR ASSISTANCE 



PLEASE RETURN THIS SURVEY AS QUICKLY AS POSSIBLE, PREFERABLY WITHIN ONE 

WEEK, IN THE ATTACHED ENVELOPE. 

Policy Studies Associates 
1718 Connecticut Avenue, NW Suite 400 
Washington, DC 20009 
Attention: Sara Nathanson 

(202) 939-9780 
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CUSTOMER SURVEY 

EISENHOWER MATH AND SCIENCE REGIONAL CONSORTIA 
PROMISING PRACTICES IN MATHEMATICS AND SCIENCE 

EDUCATION 1995 
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CUSTOMER SURVEY 

EISENHOWER MATH AND SCIENCE REGIONAL CONSORTIUMS 
"PROMISING PRACTICES IN MATHEMATICS AND SCIENCE 

EDUCATION 1995" 

Please indicate the type of agency for which you work: (CIRCLE ONE) 

a. School 1 

b. School district 2 

c. Intermediate education agency or BOCES 3 

d. State education agency 4 

e. Institution of higher education 5 

f. Other (specify): . 6 



Which of the following activities are major parts of your professional responsibilities? 
(CIRCLE ALL THAT APPLY) 

a. Teaching elementary or secondary school students 1 

b. Teaching undergraduate or graduate school students 1 

c. Administering local education program(s) or organization(s) 1 

d. Administering State education program(s) or organization(s) 1 

e. Providing professional development or technical assistance 1 

f. Other (specify): . 1 



What percent of your time is explicitly directed toward mathematics, science, or technology 
education? (CIRCLE ONE) 

a. Less than 25 percent 1 

b. 25 to 50 percent 2 

c. More than 50 percent, but less than 100 percent 3 

d. 100 percent 4 

Have you received "Promising Practices in Mathematics and Science Education 1995" from 
the [Consortium]? 

a. Yes 1 

b. No (END SURVEY AND RETURN TO PSA) 2 

c. Unsure (END SURVEY AND RETURN TO PSA) 3 



OMB No. 1880-0529. Approval expires 5/31/96. Reporting on this survey is authorized by law (20 
USC 8671). Your responses will be kept strictly confidential. Results from this survey will appear in 
summary or statistical form only, so that individuals cannot be identified. 

«9 



5 . What was the extent of your use of the document? (CIRCLE ONE) 

a. Read everything 

b. Read more than half of total 

c. Read less than half of total 

d. Read very little or nothing 



6 . What were your purposes in looking at this document? (CIRCLE ALL THAT APPLY) 



a. Generally keeping current professionally 1 

b. Obtaining ideas or contacts that would strengthen my work 1 

c. Obtaining ideas or contacts that would strengthen my 

organization’s existing work 1 

d. Obtaining ideas or contacts that would help my organization carry out a major, 

planned improvement effort 1 

e. Being a resource to others 1 

f. Other (specify): 1 



7 . How fully did the document serve your purposes? (CIRCLE ONE) 



a. It served my purposes completely 1 

b. It was a good start 2 

c. It was a start, but it did not provide sufficient information or 

guidance to enable me to follow up 3 

d. It did not serve my purposes 4 



Based on your experience, please indicate whether you think the document is a unique resource. 



a. Yes 1 

b. No 2 



If you answered "Yes" please elaborate briefly 
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Please indicate the type and extent of the follow up, if any, that the 
[Consortium] staff have initiated with you. (CIRCLE ONE ON EACH LINE) 



More than 

Never Once once 



a. 


Followed up by phone 


1 


2 


3 


b. 


Followed up with workshops, training, meetings 


1 


2 


3 


c. 


Sent additional materials 


1 


2 


3 


d. 


Visited my workplace 


1 


2 


3 


e. 


Provided financial support 


1 


2 


3 


f. 


Other (specify): 


1 


2 


3 



Have you visited the home page or other electronic databases created by this consortium, 
other Eisenhower Regional Consortiums, or the Eisenhower National Clearinghouse? 
(CIRCLE ALL THAT APPLY) 



a. Yes, I have visited the home page and/or databases created by this 

consortium 1 

b. Yes, I have visited the home page and/or databases created by other 

regional consortiums 1 

c. Yes, I have visited the home page and/or databases maintained by the 

Eisenhower National Clearinghouse 1 

d. No, I have not visited any of these electronic sites, but I am aware that they 

exist (SKIP TO 17) 1 

e. No, I have not visited any of these sites; I did not know they 

existed (SKIP TO 17) 1 



Did receiving "Promising Practices in Mathematics and Science Education 1995" 
lead you to visit one or more of these sites? 



a. Yes 1 

b. No 2 



Compared to other electronic sites you have visited, how would you rate the quality of 
these sites in each of the following areas? (CIRCLE ONE ON EACH LINE) 







Poor 


Fair 


Good 


Excellent 


a. 


Access through the Internet 


1 


2 


3 


4 


b. 


Access from one site to another 


1 


2 


3 


4 


c. 


Format of information included at the site 


1 


2 


3 


4 


d. 


Usefulness of the information at the site 


1 


2 


3 


4 
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13. 



What effect has the document had on your own work related to this topic? (CIRCLE ALL THAT 
APPLY) 



a. It confirmed what I was already doing 1 

b. I have considered doing something new in my job 1 

c. I have started to try something I learned from this activity 1 

d. I have incorporated something I learned from this activity into my job 1 

e. I have shared the ideas with a colleague(s) 1 

f. It has not had an effect on how I do my job 1 

g. Other (specify): ■ 1 



14. How large a contribution to your skills and knowledge has this document made within each of the 
following domains? (CIRCLE ONE ON EACH LINE) 







None 


Small 


Moderate 


Major 


a. 


Mathematics/science/technology education 


1 


2 


3 


4 


b. 


Professional development 


1 


2 


3 


4 


c. 


Assessment/standard setting 


1 


2 


3 


4 


d. 


Planning 


1 


2 


3 


4 


e. 


Collaboration/communication 


1 


2 


3 


4 


f. 


Other (specify): 


1 


2 


3 


4 



15. How closely is this activity related to broader (e.g., local, state, or regional) mathematics, science, 
and technology reform efforts with which you are familiar? (CIRCLE ONE) 



a. It is closely related to broader reform efforts 1 

b. It is somewhat related to broader reform efforts 2 

c. It is not related to broader reform efforts 3 

d. I don’t know; I am not involved in broader reform efforts 4 



16. What effect, if any, has the document had on the way your organization does its work? (CIRCLE 
ONE ON EACH LINE) 



None Small Moderate Major 



a. Helped generate awareness of new information 
within the organization 

b. Supported an ongoing program, policy, or practice 
of the organization 

c. Helped the organization initiate or establish a 
new program, policy, or practice 

d. Has not had an effect on the organization 



1 2 

1 2 

1 2 
1 2 



3 

3 

3 

3 



4 

4 

4 

4 
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17. Compared with other similar documents, how would you rate the quality of this one in each of the 
following areas? (CIRCLE ONE ON EACH LINE) 







Poor 


Fair 


Good 


Excellent 


a. 


Accuracy of content 


1 


2 


3 


4 


b. 


Completeness 


1 


2 


3 


4 


c. 


Presentation 


1 


2 


3 


4 


d. 


Depth 


1 


2 


3 


4 



18. Following your review of the document, how many of the promising practices sites did you contact to 
request additional information? (CIRCLE ONE) 



a. None 

b. 1-5 sites .... 

c. 6-10 sites . . . 

d. 11-15 sites . . . 

e. More than 15 sites 



1 

2 

3 

4 

5 



19. Which of the following best describes your experiences when you contacted the promising practices 
sites to request additional information? (CIRCLE ONE) 



a. There is little or no additional information available 1 

b. I was able to get information, but it was not very useful 2 

c. I was able to get a variety of useful information 3 

d. Other (specify): . 4 



THANK YOU FOR YOUR ASSISTANCE 



PLEASE RETURN THIS SURVEY AS QUICKLY AS POSSIBLE, PREFERABLY WITHIN ONE 

WEEK, IN THE ATTACHED ENVELOPE. 

Policy Studies Associates 
1718 Connecticut Avenue, NW Suite 400 
Washington, DC 20009 
Attention: Sara Nathanson 

(202) 939-9780 
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